Search NASA⌕ Search

SEARCH · Search NASA

Results for “trace anonymization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

Anonymization of Network Traces Data through Condensation-based Differential Privacy

Network traces are considered a primary source of information to researchers, who use them to investigate research problems such as identifying user behavior, analyzing network hierarchy, maintaining network security, classifying packet flows, and much more. However, most organizations are reluctant to share their data with a third party or the public due to privacy concerns. Therefore, data anonymization prior to sharing becomes a convenient solution to both organizations and researchers. Although several anonymization algorithms are available, few of them allow sufficient privacy (organization need), acceptable data utility (researcher need), and efficient data analysis at the same time. This article introduces a condensation-based differential privacy anonymization approach that achieves an improved tradeoff between privacy and utility compared to existing techniques and produces anonymized network trace data that can be shared publicly without lowering its utility value. Our solution also does not incur extra computation overhead for the data analyzer. A prototype system has been implemented, and experiments have shown that the proposed approach preserves privacy and allows data analysis without revealing the original data even when injection attacks are launched against it. When anonymized datasets are given as input to graph-based intrusion detection techniques, they yield almost identical intrusion detection rates as the original datasets with only a negligible impact.

97 MATHEMATICS AND COMPUTING↗

Using advanced data structures to enable responsive security monitoring

Write-optimized data structures (WODS), offer the potential to keep up with cyberstream event rates and give sub-second query response for key items like IP addresses. These data structures organize logs as the events are observed. To work in a real-world environment and not fill up the disk, WODS must efficiently expire older events. As the basis for our research into organizing security monitoring data, we implemented a tool, called Diventi, to index IP addresses in connection logs using RocksDB (a write-optimized LSM tree). In this work, we extended Diventi to automatically expire data as part of the data structures’ normal operations. We guarantee that Diventi always tracks the N most recent events and tracks no more than N + k events for a parameter k < N, while ensuring the index is opportunistically pruned. To test Diventi at scale in a controlled environment, we used anonymized traces of IP communications collected at SuperComputing 2019. We synthetically extended the 2.4 billion connection events to 100 billion events. We tested Diventi vs. Elasticsearch, a common log indexing tool. In our test environment, Elasticsearch saw an ingestion rate of at best 37,000 events/s while Diventi sustained ingestion rates greater than 171,000 events/s. Our query response times were as much as 100 times faster, typically answering queries in under 80 ms. Furthermore, we saw no noticeable degradation in Diventi from expiration. We have deployed Diventi for many months where it has performed well and supported new security analysis capabilities.

97 MATHEMATICS AND COMPUTING↗

PCAP Anonymizer

Explore the source record for details and available documents.

42 ENGINEERING↗

System for Secure Integration of Aviation Data

The Aviation Data Integration System (ADIS) of Ames Research Center has been established to promote analysis of aviation data by airlines and other interested users for purposes of enhancing the quality (especially safety) of flight operations. The ADIS is a system of computer hardware and software for collecting, integrating, and disseminating aviation data pertaining to flights and specified flight events that involve one or more airline(s). The ADIS is secure in the sense that care is taken to ensure the integrity of sources of collected data and to verify the authorizations of requesters to receive data. Most importantly, the ADIS removes a disincentive to collection and exchange of useful data by providing for automatic removal of information that could be used to identify specific flights and crewmembers. Such information, denoted sensitive information, includes flight data (here signifying data collected by sensors aboard an aircraft during flight), weather data for a specified route on a specified date, date and time, and any other information traceable to a specific flight. The removal of information that could be used to perform such tracing is called "deidentification." Airlines are often reluctant to keep flight data in identifiable form because of concerns about loss of anonymity. Hence, one of the things needed to promote retention and analysis of aviation data is an automated means of de-identification of archived flight data to enable integration of flight data with non-flight aviation data while preserving anonymity. Preferably, such an automated means would enable end users of the data to continue to use pre-existing data-analysis software to identify anomalies in flight data without identifying a specific anomalous flight. It would then also be possible to perform statistical analyses of integrated data. These needs are satisfied by the ADIS, which enables an end user to request aviation data associated with de-identified flight data. The ADIS includes client software integrated with other software running on flight-operations quality-assurance (FOQA) computers for purposes of analyzing data to study specified types of events or exceedences (departures of flight parameters from normal ranges). In addition to ADIS client software, ADIS includes server hardware and software that provide services to the ADIS clients via the Internet (see figure). The ADIS server receives and integrates flight and non-flight data pertaining to flights from multiple sources. The server accepts data updates from authorized sources only and responds to requests from authorized users only. In order to satisfy security requirements established by the airlines, (1) an ADIS client must not be accessible from the Internet by an unauthorized user and (2) non-flight data as airport terminal information system (ATIS) and weather data must be displayed without any identifying flight information. ADIS hardware and software architecture as well as encryption and data display scheme are designed to meet these requirements. When a user requests one or more selected aviation data characteristics associated with an event (e.g., a collision, near miss, equipment malfunction, or exceedence), the ADIS client augments the request with date and time information from encrypted files and submits the augmented request to the server. Once the user s authorization has been verified, the server returns the requested information in de-identified form.

Kulkarni, Deepak↗

Adaptive Recovery Model: Designing Systems for Testing Tracing and Vaccination to Support COVID-19 Recovery Planning.

This report documents a new approach to designing disease control policies that allocate scarce testing, contact tracing, and vaccination resources to better control community transmission of COVID19 or similar diseases. The Adaptive Recovery Model (ARM) combines a deterministic compartmental disease model with a stochastic network disease propagation model to enable us to simulate COVID-19 community spread through the lens of two complementary modeling motifs. ARM contact networks are derived from cell-phone location data that have been anonymized and interpreted as individual arrivals to specic public locations. Modeling disease spread over these networks allows us to identify locations within communities conducive to rapid disease spread. ARM applies this model- and data-derived abstractions of community transmission to evaluate the effectiveness of disease control measures including targeted social distancing, contact tracing, testing and vaccination. The architecture of ARM provides a unique capacity to help decision makers understand how best to deploy scarce testing, tracing and vaccination resources to minimize disease-spread potential in a community. This document details the novel mathematical formulations underlying ARM, presents a dynamical stability analysis of the deterministic model components, a sensitivity analysis of control parameters and network structure, and summarizes a process for deriving contact networks from cell-phone location data. An example use case steps through applying ARM to evaluate three targeted social distancing policies using Bernalillo County, New Mexico as an exemplar test locale. This step-by-step analysis demonstrates how ARM can be used to measure the relative performance of competing public health policies. Initial scenario tests of ARM shows that ARMs design focus on resource utilization rather than simple incidence prediction can provide decision makers with additional quantitative guidance for managing ongoing public health emergencies and planning future responses.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

NLR HPC Kestrel Jobs Data

Overview: Anonymized job-level records from the Kestrel HPC system at the National Laboratory of the Rockies (NLR). Each record represents a Slurm batch job with scheduling metadata, resource requests, utilization, energy estimates, and efficiency metrics. Sensitive fields (user, account, job name, submit line, working directory, submit script, and job type) are replaced with 7-character cryptographic hashes. System & Timeframe: Kestrel is located at the NLR campus. Standard compute nodes have 104 cores and 256 GB RAM; bigmem nodes have 2,000 GB. GPU nodes (gpu-h100 partition) use NVIDIA H100 GPUs. Data covers jobs submitted August 2023 through December 2025. Funding provided by the U.S. Department of Energy, EERE. Files: esif.hpc.kestrel.job-anon.zip — Anonymized job records (Hive-partitioned Parquet) datacard.md — Full dataset documentation ~11 million rows, 50 variables. Readable with PyArrow, pandas, DuckDB, Apache Spark, or any Parquet-compatible tool. Data Collection: Jobs collected via sacct with timezone-aware export (SLURM_TIME_FORMAT="%Y-%m-%dT%H:%M:%S%z"), loaded into PostgreSQL. Calculated columns updated via database triggers and batch functions. All timestamps use timestamptz and correctly handle DST transitions. Preprocessing: Anonymization of name, user, account, submit_line, work_dir, submit_script, and job_type via 7-char hex hashes Derived columns: queue_wait, cpu_eff, max/min/avg_mem_eff, energy estimates Simplified job state mapping (e.g., "CANCELLED by 132357" → "CANCELLED") Boolean flags: python_job, reframe_job Temporal decomposition: year, month, day, day_of_week, hour, minute from submit_time Shared node tracking: shared_job_count, nodes_shared, jobs_shared Key Variables: Scheduling: job_id, partition, state_simple, submit_time, start_time, end_time, queue_wait Resources: nodes_req/used, processors_req/used, memory_req, wallclock_req/used, gpus_requested Efficiency: cpu_eff, max/min/avg_mem_eff Energy: cpu_energy_tdp_estimated_max/used_watt_hours, consumed_energy_raw_joules, consumed_energy_raw_watt_hours Sharing: shared_job_count, nodes_shared, jobs_shared Partitions: short, standard, debug, gpu-h100 Job States: CANCELLED, COMPLETED, FAILED, PENDING, RUNNING QoS Levels: normal, high Important Notes: Timestamps include timezone offsets; DST transitions are handled correctly, though adding intervals across DST boundaries requires offset adjustment shared_job_count reflects physical node co-residency, not use of the shared partition Job step records and raw Slurm JSONB fields are excluded Do not attempt to re-identify individuals from hashed fields

97 MATHEMATICS AND COMPUTING↗

NLR HPC Eagle Jobs Data and Additional Energy Metrics

Overview: Anonymized job-level records from the Eagle high-performance computing (HPC) system at the National Laboratory of the Rockies (NLR). Each record represents a Slurm batch job with scheduling metadata, resource requests, resource utilization, CPU/GPU energy consumption, and efficiency metrics. Sensitive fields (user, account, job name) are replaced with cryptographic hashes. System & Timeframe: Eagle was a 2,000-node, 8-petaflop system operated at NLR from 2019–2024. Data covers the full operational lifetime of the system. Slurm data was processed nightly; timestamps are in Mountain Time. Funding provided by the U.S. Department of Energy, EERE. Files: esif.hpc.eagle.job-anon.zip — Core anonymized job records (Hive-partitioned Parquet) esif.hpc.eagle.job-anon-energy-metrics.zip — Same records with additional iLO and Ganglia energy metrics datacard.md — Full dataset documentation ~13.8 million rows, 62 variables. Readable with PyArrow, pandas, DuckDB, Apache Spark, or any Parquet-compatible tool. Data Collection: Jobs collected via sacct through a pipeline: Eagle Jobs API → Redpanda → StreamSets → HPCMON API → PostgreSQL. Node-level power from iLO (HP Integrated Lights-Out); GPU power from Ganglia monitoring, joined to jobs via node lists and time ranges. Preprocessing: Anonymization of name, user, and account fields via cryptographic hashing Derived columns: queue_wait, cpu_eff, max_mem_eff Simplified job state mapping (e.g., "CANCELLED BY 12345" → "CANCELLED") QoS accounting rules (buy-in, standby, or Slurm QoS value) CPU energy estimated from TDP (200W, Intel Xeon Gold 6154, 18 cores) Timezone-aware columns (_tz) sourced from LEX accounting database to correctly handle DST transitions Key Variables: Scheduling: job_id, partition, state_simple, submit_time_tz, start_time_tz, end_time_tz, queue_waitResources: nodes_req/used, processors_req/used, memory_req, wallclock_req/used, gpus_requested Efficiency: cpu_eff, max_mem_eff Energy: cpu_energy_tdp_estimated_max/used_watt_hours, node_energy_total_watt_hours (iLO), gpu0/1_energy_total_watt_hours (Ganglia) Partitions: bigmem, bigmem-8600, bigscratch, csc, dav, ddn, debug, gpu, haswell, long, mono, short, standard Job States: CANCELLED, COMPLETED, FAILED, NODE_FAIL, OUT_OF_MEMORY, PENDING, RUNNING, TIMEOUT QoS Levels: Unknown, normal, buy-in, debug, penalty, high, standby Important Notes: Non-_tz timestamp columns may be off by one hour across DST boundaries; use _tz columns for time difference calculations Energy fields are null for jobs without monitoring coverage Job step records and raw Slurm JSONB fields are excluded from this extract Do not attempt to re-identify individuals from hashed fields

97 MATHEMATICS AND COMPUTING↗