SEARCH · Search NASA

Results for “Workflow pattern detection”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

23 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2026

NMF-Based Anomaly Detection in CMS 2D Tracking Occupancy Histograms

The CMS experiment relies on Data Quality Monitoring (DQM) to ensure that recorded collision data are suitable for physics analysis. During LHC Run 3, each run contains many lumisections and tracking monitoring elements, making offline inspection challenging, especially for localized detector effects that may appear only for short periods of time. This poster presents an unsupervised machine-learning approach to identify anomalous lumisections in CMS tracking occupancy histograms using Non-Negative Matrix Factorization (NMF). The workflow uses offline CMS DQMIO tracking histograms retrieved with the CMS DIALS API and organized as two-dimensional occupancy maps for each lumisection. After selecting stable lumisections, the occupancy maps are normalized and arranged into a non-negative data matrix. The NMF model learns a compact set of basis patterns describing normal tracking occupancy. Each lumisection is then reconstructed from these learned components, and the reconstruction error is used as an anomaly score. Large residuals indicate occupancy patterns that deviate from normal detector behavior and are flagged for further inspection. This NMF-based approach provides a fast and interpretable way to flag lumisections whose tracking occupancy patterns differ from normal detector behavior. Preliminary studies show sensitivity to known tracking anomalies, and ongoing work is focused on validating the method across additional Run 3 Pixel and Strip detector issues.

Rodríguez Ramos, Iliomar [Puerto Rico U., Mayaguez↗

DOE OSTI2024

Snow Distribution Patterns Revisited: A Physics-Based and Machine Learning Hybrid Approach to Snow Distribution Mapping in the Sub-Arctic

Snowpack distribution in Arctic and alpine landscapes often occurs in repeating, year-to-year patterns due to local topographic, weather, and vegetation characteristics. Previous studies have suggested that with years of observational data, these snow distribution patterns can be statistically integrated into a snow process modeling workflow. Recent advances in snow hydrology and machine learning (ML) have increased our ability to predict snowpack distribution using in-situ observations, remote sensing data sets, and simple landscape characteristics that can be easily obtained for most environments. Here, we propose a hybrid approach to couple a ML snow distribution pattern (MLSDP) map with a physics-based, snow process model. We trained a random forest ML algorithm on tens of thousands of snow survey observations from a subarctic study area on the Seward Peninsula, Alaska, collected during peak snow water equivalent (SWE). We validated hybrid model outputs using in-situ snow depth and SWE observations, as well as a light detection and ranging data set and a distributed temperature profiling sensor data set. When the hybrid results were compared with the physics-based method, the hybrid method more accurately depicted the spatial patterns of the snowpack, areas of drifting snow, and years when no in-situ observations were used in the random forest ML training data set. The hybrid method also showed improvements in root mean squared error at 61% of locations where time-series estimations of snow depth were observed. These results can be applied to any physics-based model to improve the snow distribution patterning to reflect observed conditions in high latitude and high elevation cold region environments.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

Analyzing Risks of Virtual Private Network Connections

The use of Splunk for analyzing VPN logs is an effective approach for identifying vulnerabilities in network endpoints. Splunk, a powerful platform for searching, monitoring, and analyzing machine-generated data, enables organizations to aggregate VPN logs in real-time, providing insights into network activity, user behavior, and potential security risks. By indexing VPN traffic and authentication logs, security teams can track abnormal patterns such as multiple failed login attempts, unusual IP addresses, or unexpected changes in bandwidth usage, all of which could indicate potential vulnerabilities or breaches. With Splunk’s advanced search and reporting capabilities, users can create custom dashboards and alerts to detect suspicious activities. Automated searches can flag endpoints exhibiting unusual behavior, while correlation analysis can identify links between compromised devices and broader network vulnerabilities. In particular, Splunk's machine learning capabilities can be leveraged to predict and prevent threats by identifying trends that might otherwise be missed in traditional log analysis. This proactive approach to monitoring VPN logs allows for the early detection of security weaknesses, enabling rapid response and minimizing potential damage to network integrity. By enhancing endpoint visibility, Splunk plays a crucial role in securing remote connections and safeguarding sensitive information. Additionally, Splunk’s automation and alerting features allow teams to create custom workflows that notify them of vulnerable or misconfigured endpoints identified through Shodan. This synergy between Splunk’s log analysis and Shodan’s device intelligence enhances an organization’s ability to proactively identify and mitigate security risks, improving the overall resilience of their VPN infrastructure.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows

This report details recent progress for the ASCR funded project “Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows”. We refer to the project as IPPD/2, reflecting the 2017 renewal under expanded scope and partners In IPPD/2, we increased our research scope to include data motion. We are focusing on three major aspects: a) observe how data is generated, distributed, and used; b) analyze how data is (repeatedly) consumed with a focus both on repeated patterns and anomalies; and c) explore how to optimize data motion. This new work on data motion will augment and complement IPPD/2’s research that focused on the computational aspects of tasks. We leverage and extend our existing tools and demonstrate our work on the Belle II workflow suite as well as on workflows from NSLS-II. The highlights of our work are as follows: Provenance for Workflows: Provenance is used to provide information enabling quality control, re-run computational workflows, and reproduce results. IPPD/2 has been building a scalable provenance management system that enables the capture of provenance from the high-level workflow through all relevant system levels in one integrated environment. Leveraging this work, our recent efforts have included using provenance as an enabling technique. Workload characterization: Leveraging provenance and analysis, we characterize data movement within network, storage, and memory over a variety of workloads. This characterization enables an understanding by performance analysts and application developers of the range of behaviors that could be expected. Performance Prediction for Workflows: The goal of modeling distributed workflows is to understand performance bottlenecks and enable more intelligent task scheduling to optimize selected metrics of interest (e.g., task throughput or output data rate). IPPD/2 has utilized both analytical and AI/ML modeling methodologies for performance modeling. Advanced Scheduling and Fault Modeling for Workflows: Scheduling of large-scale scientific workflows on geographically distributed resources is a challenging problem. To improve workflow throughput, we combined novel scheduling algorithms with task predictions from performance modeling and fault modeling. Dynamically Alleviating Bottlenecks in Workflows: Exploiting our provenance, analysis, and modeling efforts, we have explored and developed several techniques for dynamically detecting and alleviating bottlenecks in data movement. In particular, we have spent considerable effort demonstrating our techniques on production-like workflow configurations.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Expanding standards in viromics: in silico evaluation of dsDNA viral genome identification, classification, and auxiliary metabolic gene curation

Viruses influence global patterns of microbial diversity and nutrient cycles. Though viral metagenomics (viromics), specifically targeting dsDNA viruses, has been critical for revealing viral roles across diverse ecosystems, its analyses differ in many ways from those used for microbes. To date, viromics benchmarking has covered read pre-processing, assembly, relative abundance, read mapping thresholds and diversity estimation, but other steps would benefit from benchmarking and standardization. Here we use in silico-generated datasets and an extensive literature survey to evaluate and highlight how dataset composition (i.e., viromes vs bulk metagenomes) and assembly fragmentation impact (i) viral contig identification tool, (ii) virus taxonomic classification, and (iii) identification and curation of auxiliary metabolic genes (AMGs). The in silico benchmarking of five commonly used virus identification tools show that gene-content-based tools consistently performed well for long (≥3 kbp) contigs, while k -mer- and blast-based tools were uniquely able to detect viruses from short (≤3 kbp) contigs. Notably, however, the performance increase of k -mer- and blast-based tools for short contigs was obtained at the cost of increased false positives (sometimes up to ~5% for virome and ~75% bulk samples), particularly when eukaryotic or mobile genetic element sequences were included in the test datasets. Furthermore, for viral classification, variously sized genome fragments were assessed using gene-sharing network analytics to quantify drop-offs in taxonomic assignments, which revealed correct assignations ranging from ~95% (whole genomes) down to ~80% (3 kbp sized genome fragments). A similar trend was also observed for other viral classification tools such as VPF-class, ViPTree and VIRIDIC, suggesting that caution is warranted when classifying short genome fragments and not full genomes. Finally, we highlight how fragmented assemblies can lead to erroneous identification of AMGs and outline a best-practices workflow to curate candidate AMGs in viral genomes assembled from metagenomes. Together, these benchmarking experiments and annotation guidelines should aid researchers seeking to best detect, classify, and characterize the myriad viruses ‘hidden’ in diverse sequence datasets.

59 BASIC BIOLOGICAL SCIENCES↗