Search NASA⌕ Search

SEARCH · Search NASA

Results for “PostgreSQL”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Database Performance Monitoring for DUNE

This report presents the research, design, and implementation of improved PostgreSQL monitoring for DUNE Rucio database services using Checkmk. The project began with a request to improve dashboard visibility for database performance metrics, including connection usage, configured connection limits, lock activity, wait behavior, storage trends, query performance, and saturation alerts. The initial implementation focused on the dune_rucio_prod database on the rucio_prod PostgreSQL instance because connection saturation and lock contention are direct reliability risks for database-backed services. Existing Checkmk PostgreSQL monitoring was investigated, and several gaps were identified. Built-in connection monitoring did not clearly separate active, idle, idle-in-transaction, total, and usage-percent metrics, while the built-in lock monitoring simplified PostgreSQL lock modes into shared and exclusive categories. To address these gaps, two DSG-specific Checkmk local checks were created: one for connection-state monitoring and one for lock-state monitoring. These checks supplement the built-in PostgreSQL checks and provide additional performance data for dashboard graphs, service states, and alerts.

Bowers, Elliot [Cabrillo Coll.]↗
DOE OSTI2026

Expandable Log Analyzing Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, IL]↗
DOE OSTI2026

Expandable Log Analyzing Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, IL]↗
DOE OSTI2026

Database Performance Monitoring for DUNE

This project improves Checkmk monitoring for DUNE Rucio PostgreSQL database services by adding clearer dashboard visibility for connection and lock behavior. The work began with a request to monitor database performance metrics such as connection usage, configured limits, lock activity, wait behavior, query performance, storage trends, and saturation alerts. Existing Checkmk PostgreSQL checks were reviewed, and gaps were identified in how connection states and lock modes were displayed. To address these gaps, two DSG-specific local checks were added for dune_rucio_prod: one for connection-state monitoring and one for lock-state monitoring. These checks report active, idle, idle-in-transaction, total, usage-percent, lock-mode, waiting-lock, and wait-age metrics. The added metrics supplement built-in Checkmk monitoring and provide DUNE application developers with clearer service states, history graphs, dashboard widgets, and alerts.

Bowers, Elliot [Cabrillo Coll.]↗
DOE OSTI2026

Database-Agnostic Log Analysis and Monitoring Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, US, IL; Fermilab]↗
DOE OSTI2020

Report on the deployment of the National Geothermal Data System 2.0

This reports includes a video description of recent upgrades and changes to the National Geothermal Data System (geothermaldata.org) and a text report of its relevant security upgrades. Improvements include a new operating system, implementation of HTTPS, implementation of a standard firewall, PostgreSQL upgrades, an ESRI ArcGIS server, new registration policies, and a non-public API.

15 GEOTHERMAL ENERGY↗
DOE OSTI2020

High Resolution Data Analysis: Plans and Prospects [Book Chapter]

Herein, a report on the progress on the high resolution data analysis of the ADMX experimental results is presented. In this paper, tools are developed and tested on a blind injection mimicking a Maxwellian like signal in the frequency domain. This blind injection will be used as a test bed which can be later implemented on all the high resolution data. The high resolution data is stored in the Fermilab server. In this analysis a PostgreSQL query was made to ensure the blind injection is in the middle of the frequency spectrum and 19 such files were found. The time series data is read using a c++ program. An apodization function is applied on the time series data and zero filled to reduce the frequency spacing in order to achieve a better interpolation. A FFTW header is used to compute the Fourier transform of the time series data. A Savitzky–Golay filter is applied on the unnormalized power which then can be used to remove the spectral shape. Each frequency spectrum has a bandwidth of 50 kHz.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2026

SoK: What does it Mean to Benchmark Database Forensics?

Relational Database Management Systems are the backbone of modern enterprises and public-sector services, and are thus frequent targets of security incidents, insider threats, and thorough regulatory audits. Consequently, databases have become key sources of digital evidence, requiring investigators to reconstruct past activity from audit logs, transaction logs, and backups. Although benchmarking frameworks such as those developed by the Transaction Processing Performance Council (TPC) are widely used to evaluate database performance, they do not capture forensic requirements such as evidentiary completeness, tamper-evidence, chain of custody, or regulatory compliance under GDPR and CCPA. This survey examines the emerging domain of forensic database benchmarking. We gathered prior research on database forensics, secure logging, and tamper-evident data structures; we analyze modern forensic-ready features in commercial and open-source systems (SQL Server Ledger, Oracle Blockchain Tables, PostgreSQL pgAudit, Db2 Audit, Aurora Database Activity Streams, Oracle Real Application Security and IBM Guardium) and assess why existing benchmarks are insufficient. We propose forensic workloads, metrics, and methodologies that incorporate adversarial stressors, deleted-record recovery, and backup analysis. We also identify open research problems and call for a community-driven forensic benchmark suite. The result is an idea for evaluating not only database performance but also forensic soundness, bridging the gap between system engineering, compliance, and digital investigations.

Lenard, Ben↗
DOE OSTI2021

Can Applications Recover from fsync Failures?

We analyze how file systems and modern data-intensive applications react to fsync failures. First, we characterize how three Linux file systems (ext4, XFS, Btrfs) behave in the presence of failures. We find commonalities across file systems (pages are always marked clean, certain block writes always lead to unavailability) as well as differences (page content and failure reporting is varied). Next, we study how five widely used applications (PostgreSQL, LMDB, LevelDB, SQLite, Redis) handle fsync failures. Our findings show that although applications use many failure-handling strategies, none are sufficient: fsync failures can cause catastrophic outcomes such as data loss and corruption. Our findings have strong implications for the design of file systems and applications that intend to provide strong durability guarantees.

Computer Science↗
DOE OSTI2021

webspinner

Python utilities for working with data source types used by NREL's dsgrid (Demand-Side Grid Model) project (i.e. AWS, PostgreSQL, and .parquet). https://www.nrel.gov/analysis/dsgrid.html

Hale, Elaine↗
DOE OSTI2018

pnnl/pg-histogram

PostgreSQL histograms for ActiveRecord relations. A plugin for the Ruby on Rails (RoR) web application development framework web application development framework that leverages the "Active Relation" relational algebra component to automatically formulate SQL statements for the construction of histograms.

Borkum, Mark↗
DOE OSTI2026

datasight [SWR-26-045]

This software is an AI-powered data exploration with natural language. datasight connects an AI agent to your database and provides a web UI where you can ask questions in natural language. The agent writes SQL, runs queries, and generates interactive Plotly visualizations. Supports DuckDB, PostgreSQL, SQLite, and Flight SQL databases. Also queries local CSV and Parquet files directly — no database setup required. Supports Anthropic Claude (default), GitHub Models (open source), and Ollama (local) as LLM backends.

Thom, Daniel [National Laboratory of the Rockies (↗
DOE OSTI2026

AQDrop Quantum Service (AQDrop) v1.0

AQDrop is a job management system designed to streamline access to the Advanced Quantum Testbed (AQT) at NERSC (National Energy Research Scientific Computing Center). It serves as a centralized middleware layer between researchers and quantum processing hardware. Key Features: AQDrop provides a FastAPI-based server backed by PostgreSQL for job submission, queue management, and role-based access control (members, operators, and administrators). Users submit Qiskit circuits via JSON payloads, which are queued, dispatched to the QPU through the Qubic API, and returned as measurement counts. A Python client library and web dashboard round out the interface options. Primary Use: Researchers submit quantum circuit jobs from a laptop or login node; an operator client executes those jobs on the AQT's physical QPU and returns results — all coordinated through the central API. Advantages: Compared to ad-hoc or direct hardware access, AQDrop adds structured queue management, auditable job-status tracking and OAuth2 authentication — reducing scheduling conflicts and unauthorized access. Its containerized deployment also improves reproducibility and scalability. Overall, AQDrop functions as a purpose-built quantum job broker tailored to NERSC's specific hardware and institutional access requirements.

Caplinger, Evan [Lawrence Berkeley National Labora↗
DOE OSTI2019

Open MPI Testing Infrastructure Update - September, 2019

Since the last update on the Open MPI regression testing infrastructure, and the deployment of the MTT Python client on more DOE HPC centers, problems with the client and its interaction with the community PostGreSQL database server at AWS began to appear. In particular, problems were found with the reporting of failed and timed out tests to the community database.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Rucio at LSST/Rubin

In this presentation, we will explore the Rucio experience with the Rubin Observatory experiment. Our discussion will cover several key areas: Scalability Tests: Insights into the performance and scalability evaluations of Rucio in the context of Rubin's data needs and what we have learned, especially with many small files. Role in Rubin's Data Curation: Rubin's Data Butler: An overview of how Rucio, along with with Rubin's Data Butler using Hermes-K, which involves message passing through Kafka, is integrated in the Rubin's data curation system. Monitoring and Support: Current status of Rucio and PostgreSQL monitoring and Rucio deployment and support within the Rubin environment. Tape RSE Implementation: Deal with the order of magnitude more files going to tape than HEP. Future Needs: An examination of Rubin's evolving requirements for Rucio services and how we plan to address them.

Lee, Dennis [Fermilab]↗
DOE OSTI2020

dGen (Distributed Generation Market Demand) Model Data: Alpha Release

Open sourced data needed to run the basic alpha release version of the dGen model. Includes a pre-generated agent file of 100,000 agents in pickle file format along with the base schema and table data in parquet format that are needed to create a postgreSQL database for the model to interact with.

14 SOLAR ENERGY↗
DOE OSTI2025

NLR HPC Kestrel Jobs Data

Overview: Anonymized job-level records from the Kestrel HPC system at the National Laboratory of the Rockies (NLR). Each record represents a Slurm batch job with scheduling metadata, resource requests, utilization, energy estimates, and efficiency metrics. Sensitive fields (user, account, job name, submit line, working directory, submit script, and job type) are replaced with 7-character cryptographic hashes. System & Timeframe: Kestrel is located at the NLR campus. Standard compute nodes have 104 cores and 256 GB RAM; bigmem nodes have 2,000 GB. GPU nodes (gpu-h100 partition) use NVIDIA H100 GPUs. Data covers jobs submitted August 2023 through December 2025. Funding provided by the U.S. Department of Energy, EERE. Files: esif.hpc.kestrel.job-anon.zip — Anonymized job records (Hive-partitioned Parquet) datacard.md — Full dataset documentation ~11 million rows, 50 variables. Readable with PyArrow, pandas, DuckDB, Apache Spark, or any Parquet-compatible tool. Data Collection: Jobs collected via sacct with timezone-aware export (SLURM_TIME_FORMAT="%Y-%m-%dT%H:%M:%S%z"), loaded into PostgreSQL. Calculated columns updated via database triggers and batch functions. All timestamps use timestamptz and correctly handle DST transitions. Preprocessing: Anonymization of name, user, account, submit_line, work_dir, submit_script, and job_type via 7-char hex hashes Derived columns: queue_wait, cpu_eff, max/min/avg_mem_eff, energy estimates Simplified job state mapping (e.g., "CANCELLED by 132357" → "CANCELLED") Boolean flags: python_job, reframe_job Temporal decomposition: year, month, day, day_of_week, hour, minute from submit_time Shared node tracking: shared_job_count, nodes_shared, jobs_shared Key Variables: Scheduling: job_id, partition, state_simple, submit_time, start_time, end_time, queue_wait Resources: nodes_req/used, processors_req/used, memory_req, wallclock_req/used, gpus_requested Efficiency: cpu_eff, max/min/avg_mem_eff Energy: cpu_energy_tdp_estimated_max/used_watt_hours, consumed_energy_raw_joules, consumed_energy_raw_watt_hours Sharing: shared_job_count, nodes_shared, jobs_shared Partitions: short, standard, debug, gpu-h100 Job States: CANCELLED, COMPLETED, FAILED, PENDING, RUNNING QoS Levels: normal, high Important Notes: Timestamps include timezone offsets; DST transitions are handled correctly, though adding intervals across DST boundaries requires offset adjustment shared_job_count reflects physical node co-residency, not use of the shared partition Job step records and raw Slurm JSONB fields are excluded Do not attempt to re-identify individuals from hashed fields

97 MATHEMATICS AND COMPUTING↗