Search NASA⌕ Search

SEARCH · Search NASA

Results for “bottleneck detection”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis

Using parallel file systems efficiently is a tricky problem due to inter-dependencies among multiple layers of I/O software, including high-level I/O libraries (HDF5, netCDF, etc.), MPI-IO, POSIX, and file systems (GPFS, Lustre, etc.). Profiling tools such as Darshan collect traces to help understand the I/O performance behavior. However, there are significant gaps in analyzing the collected traces and then applying tuning options offered by various layers of I/O software. Seeking to connect the dots between I/O bottleneck detection and tuning, we propose DXT Explorer, an interactive log analysis tool. In this paper, we present a case study using our interactive log analysis tool to identify and apply various I/O optimizations. We report an evaluation of performance improvement achieved for four I/O kernels extracted from science applications.

Bez, Jean Luca↗

Bottleneck Detection in Modular Construction Factories Using Computer Vision

The construction industry is increasingly adopting off-site and modular construction methods due to the advantages offered in terms of safety, quality, and productivity for construction projects. Despite the advantages promised by this method of construction, modular construction factories still rely on manually-intensive work, which can lead to highly variable cycle times. As a result, these factories experience bottlenecks in production that can reduce productivity and cause delays to modular integrated construction projects. To remedy this effect, computer vision-based methods have been proposed to monitor the progress of work in modular construction factories. However, these methods fail to account for changes in the appearance of the modular units during production, they are difficult to adapt to other stations and factories, and they require a significant amount of annotation effort. Due to these drawbacks, this paper proposes a computer vision-based progress monitoring method that is easy to adapt to different stations and factories and relies only on two image annotations per station. In doing so, the Scale-invariant feature transform (SIFT) method is used to identify the presence of modular units at workstations, and the Mask R-CNN deep learning-based method is used to identify active workstations. This information was synthesized using a near real-time data-driven bottleneck identification method suited for assembly lines in modular construction factories. This framework was successfully validated using 420 h of surveillance videos of a production line in a modular construction factory in the U.S., providing 96% accuracy in identifying the occupancy of the workstations and an F-1 Score of 89% in identifying the state of each station on the production line. The extracted active and inactive durations were successfully used via a data-driven bottleneck detection method to detect bottleneck stations inside a modular construction factory. The implementation of this method in factories can lead to continuous and comprehensive monitoring of the production line and prevent delays by timely identification of bottlenecks.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Advanced I/O for large-scale scientific applications.

As scientific simulations scale to use petascale machines and beyond, the data volumes generated pose a dual problem. First, with increasing machine sizes, the careful tuning of IO routines becomes more and more important to keep the time spent in IO acceptable. It is not uncommon, for instance, to have 20% of an application's runtime spent performing IO in a 'tuned' system. Careful management of the IO routines can move that to 5% or even less in some cases. Second, the data volumes are so large, on the order of 10s to 100s of TB, that trying to discover the scientifically valid contributions requires assistance at runtime to both organize and annotate the data. Waiting for offline processing is not feasible due both to the impact on the IO system and the time required. To reduce this load and improve the ability of scientists to use the large amounts of data being produced, new techniques for data management are required. First, there is a need for techniques for efficient movement of data from the compute space to storage. These techniques should understand the underlying system infrastructure and adapt to changing system conditions. Technologies include aggregation networks, data staging nodes for a closer parity to the IO subsystem, and autonomic IO routines that can detect system bottlenecks and choose different approaches, such as splitting the output into multiple targets, staggering output processes. Such methods must be end-to-end, meaning that even with properly managed asynchronous techniques, it is still essential to properly manage the later synchronous interaction with the storage system to maintain acceptable performance. Second, for the data being generated, annotations and other metadata must be incorporated to help the scientist understand output data for the simulation run as a whole, to select data and data features without concern for what files or other storage technologies were employed. All of these features should be attained while maintaining a simple deployment for the science code and eliminating the need for allocation of additional computational resources.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

A Zoom-in Analysis of I/O Logs to Detect Root Causes of I/O Performance Bottlenecks

Scientific applications frequently spend a large fraction of their execution time in reading and writing data on parallel file systems. Identifying these I/O performance bottlenecks and attributing root causes are critical steps toward devising optimization strategies. Several existing studies analyze I/O logs of a set of benchmarks or applications that were run with controlled behaviors. However, there is still a lack of general approach that systematically identifies I/O performance bottlenecks for applications running "in the wild" on production systems. In this study, we have developed an analysis approach of "zooming in" from platform-wide to application-wide to job-level I/O logs for identifying I/O bottlenecks in arbitrary scientific applications. We analyze the logs collected on a Cray XC40 system in production over a two-month period. This study results in several insights for application developers to use in optimizing I/O behavior.

Wang, Teng↗

Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows

This report details recent progress for the ASCR funded project “Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows”. We refer to the project as IPPD/2, reflecting the 2017 renewal under expanded scope and partners In IPPD/2, we increased our research scope to include data motion. We are focusing on three major aspects: a) observe how data is generated, distributed, and used; b) analyze how data is (repeatedly) consumed with a focus both on repeated patterns and anomalies; and c) explore how to optimize data motion. This new work on data motion will augment and complement IPPD/2’s research that focused on the computational aspects of tasks. We leverage and extend our existing tools and demonstrate our work on the Belle II workflow suite as well as on workflows from NSLS-II. The highlights of our work are as follows: Provenance for Workflows: Provenance is used to provide information enabling quality control, re-run computational workflows, and reproduce results. IPPD/2 has been building a scalable provenance management system that enables the capture of provenance from the high-level workflow through all relevant system levels in one integrated environment. Leveraging this work, our recent efforts have included using provenance as an enabling technique. Workload characterization: Leveraging provenance and analysis, we characterize data movement within network, storage, and memory over a variety of workloads. This characterization enables an understanding by performance analysts and application developers of the range of behaviors that could be expected. Performance Prediction for Workflows: The goal of modeling distributed workflows is to understand performance bottlenecks and enable more intelligent task scheduling to optimize selected metrics of interest (e.g., task throughput or output data rate). IPPD/2 has utilized both analytical and AI/ML modeling methodologies for performance modeling. Advanced Scheduling and Fault Modeling for Workflows: Scheduling of large-scale scientific workflows on geographically distributed resources is a challenging problem. To improve workflow throughput, we combined novel scheduling algorithms with task predictions from performance modeling and fault modeling. Dynamically Alleviating Bottlenecks in Workflows: Exploiting our provenance, analysis, and modeling efforts, we have explored and developed several techniques for dynamically detecting and alleviating bottlenecks in data movement. In particular, we have spent considerable effort demonstrating our techniques on production-like workflow configurations.

97 MATHEMATICS AND COMPUTING↗

Scalable Comparative Visualization of Ensembles of Call Graphs

Optimizing the performance of large-scale parallel codes is critical for efficient utilization of computing resources. Code developers often explore various execution parameters, such as hardware configurations, system software choices, and application parameters, and are interested in detecting and understanding bottlenecks in different executions. They often collect hierarchical performance profiles represented as call graphs, which combine performance metrics with their execution contexts. The crucial task of exploring multiple call graphs together is tedious and challenging because of the many structural differences in the execution contexts and significant variability in the collected performance metrics (e.g., execution runtime). In this paper, we present Ensemble CallFlow to support the exploration of ensembles of call graphs using new types of visualizations, analysis, graph operations, and features. We introduce ensemble-Sankey , a new visual design that combines the strengths of resource-flow (Sankey) and box-plot visualization techniques. Whereas the resource-flow visualization can easily and intuitively describe the graphical nature of the call graph, the box plots overlaid on the nodes of Sankey convey the performance variability within the ensemble. Our interactive visual interface provides linked views to help explore ensembles of call graphs, e.g., by facilitating the analysis of structural differences, and identifying similar or distinct call graphs. Finally, we demonstrate the effectiveness and usefulness of our design through case studies on large-scale parallel codes.

97 MATHEMATICS AND COMPUTING↗

DXT Explorer v0.1

DTX Explorer is a tool to generate interactive data visualizations of Darshan I/O traces collected from HPC applications. Its goal is to provide an easy and interactive way for researchers and developers to explore their application's I/O behavior and detect possible I/O bottlenecks that are impacting performance.

Bez, Jean Luca↗

DXT Explorer v2.0

DTX Explorer is a tool to generate interactive data visualizations of Darshan I/O traces collected from HPC applications. Its goal is to provide an easy and interactive way for researchers and developers to explore their application's I/O behavior and detect possible I/O bottlenecks that are impacting performance.

Bez, JeanLuca↗

Speeding up low-mass planetary microlensing simulations and modeling: The caustic region of influence

Extensive simulations of planetary microlensing are necessary both before and after a survey is conducted: before to design and optimize the survey and after to understand its detection efficiency. The major bottleneck in such computations is the computation of light curves. However, for low-mass planets, most of these computations are wasteful, as most light curves do not contain detectable planetary signatures. In this paper, I develop a parameterization of the binary microlens that is conducive to avoiding light curve computations. I empirically find analytic expressions describing the limits of the parameter space that contain the vast majority of low-mass planet detections. Through a large-scale simulation, I measure the (in)completeness of the parameterization and the speed-up it is possible to achieve. For Earth-mass planets in a wide range of orbits, it is possible to speed up simulations by a factor of ∼30-125 (depending on the survey's annual duty-cycle) at the cost of missing ∼1% of detections (which is actually a smaller loss than for the arbitrary parameter limits typically applied in microlensing simulations). The benefits of the parameterization probably outweigh the costs for planets below 100 M{sub ⊕}. For planets at the sensitivity limit of AFTA-WFIRST, simulation speed-ups of a factor ∼1000 or more are possible.

79 ASTRONOMY AND ASTROPHYSICS↗

Using Phage and Yeast Display to Select Hundreds of Monoclonal Antibodies: Application to Antigen 85, a Tuberculosis Biomarker

Background: Current diagnostic methods for tuberculosis (TB), a major global health challenge that kills nearly two million people annually, are time-consuming and inadequate. During infection a number of bacterial molecules that play a role in the infective process are released and have been proposed as biomarkers for early TB diagnosis. Antigen 85 (Ag85) is the most abundant secreted TB protein, and a potential target for this diagnostic approach. One of the bottlenecks in the direct detection of such bacterial targets is the availability of robust, sensitive, specific antibodies. Methods: Using Ag85 as a model, we describe a method to select antibodies against any potential target using a novel combination of phage and yeast display that exploits the advantage of each approach. Results: The efficiency of this approach was attested to by the 111 specific antibodies identified in initial screens. These were assessed for binding to the different Ag85 subunits, affinity, and activity in sandwich assays. Conclusions: The novelty of this approach lies in the possibility of screening the entire output of a phage antibody selection in a single experiment by yeast display. This can be considered analogous to carrying out a million ELISAs. The monoclonal antibodies (mAbs) identified in this way show high binding affinity and selectivity for the antigens and offer an advantage over traditional mAbs produced by relatively expensive and time consuming techniques. This approach has wide applicability, and the affinity of selected antibodies can be significantly improved, if required.

Antibodies↗

Giant isotope effect on phonon dispersion and thermal conductivity in methylammonium lead iodide

Lead halide perovskites are strong candidates for high-performance low-cost photovoltaics, light emission, and detection applications. A hot-phonon bottleneck effect significantly extends the cooling time of hot charge carriers, which thermalize through carrier–optic phonon scattering, followed by optic phonon decay to acoustic phonons and finally thermal conduction. To understand these processes, we adjust the lattice dynamics independently of electronics by changing isotopes. We show that doubling the mass of hydrogen in methylammonium lead iodide by replacing protons with deuterons causes a large 20 to 50% softening of the longitudinal acoustic phonons near zone boundaries, reduces thermal conductivity by ~50%, and slows carrier relaxation kinetics. Phonon softening is attributed to anticrossing with the slowed libration modes of the deuterated molecules and the reduced thermal conductivity to lowered phonon velocities. Our results reveal how tuning the organic molecule dynamics enables control of phonons important to thermal conductivity and the hot-phonon bottleneck.

36 MATERIALS SCIENCE↗

Bacteriophage Tail-Tube Assembly Studied by Proton-Detected 4D Solid-State NMR

Obtaining unambiguous resonance assignments remains a major bottleneck in solid-state NMR studies of protein structure and dynamics. Particularly for supramolecular assemblies with large subunits (>150 residues), the analysis of crowded spectral data presents a challenge, even if three-dimensional (3D) spectra are used. Here, we present a proton-detected 4D solid-state NMR assignment procedure that is tailored for large assemblies. The key to recording 4D spectra with three indirect carbon or nitrogen dimensions with their inherently large chemical shift dispersion lies in the use of sparse non-uniform sampling (as low as 2 %). As a proof of principle, we acquired 4D (H)COCANH, (H)CACONH, and (H)CBCANH spectra of the 20 kDa bacteriophage tail-tube protein gp17.1 in a total time of two and a half weeks. These spectra were sufficient to obtain complete resonance assignments in a straightforward manner without use of previous solution NMR data.

59 BASIC BIOLOGICAL SCIENCES↗

Mitigate: An Adaptive Network Data Anonymization Tool Using Condensation-Based Differential Privacy

Modern network devices collect a large amount of data that can be analyzed to identify bottlenecks, anomalies, cyber-attacks, etc. Therefore, there is often a need to analyze such collections of network data quite often by an external expert or by the research community. However, these collections of data contain sensitive, proprietary information. In order for the network data to be shared, it must first be anonymized. The overall objective of this project is to develop an innovative privacy management tool to anonymize network data and achieve sufficient privacy, acceptable data utility, and efficient data analysis at the same time. No existing anonymization methods can achieve all of these at the same time. The core of this technology is a differential private clustering algorithm that provides strong privacy protection, preserves data properties important for subsequent analysis, and allows the party receiving the anonymized data to conduct analysis directly on anonymized data without the need of decryption or any extra processing. The research carried out was to design, implement and verify a solution to this problem by completing the following tasks: 1) developing the core technology; 2) developing a context based method that automatically recommends fields that must be anonymized; 3) conducted experiments showing superior results using our approach compared to existing tools, and 4) developed an intuitive but basic user interface. The research that was conducted generated novel algorithmic techniques that utilize state-of-the-art methods such as condensation, differential privacy preservation, clustering, automated tuning based on contextual awareness, and recommendation techniques to specify columns to users for anonymization leading to optimal privacy that allows research analysis on the dataset. Experiments were conducted to evaluate the efficacy of these novel algorithmic techniques by performing analysis on original non-anonymized datasets, then conducting analysis on the same yet anonymized datasets and comparing the results of the analyses. Overall, the anonymized analysis results were within 1% of the original results, verifying that the generated technology not only guarantees a high level of privacy but also enables research analysis as if it were conducted on the original dataset. Potential applications of this technology include anonymization of any type of structured network datasets that contain sensitive identifiers, such as IP addresses, that can be used in multiple applications. For example, to create an AI or machine learning model for cyber security, e.g., to detect attacks, or for performance analysis, e.g., identify bottlenecks or predict performance. In addition, a market analysis that was conducted for potential applications of this technology identified a broader range of applications of our anonymization technology beyond the network sector that includes healthcare, banking, insurance, securities, finance (FISB), data brokering, cloud services, ad sales, and government.

97 MATHEMATICS AND COMPUTING↗

Pipelining RDP Data to the "Taxomatic" (Final Report)

This project builds on the results of previously funded research by integrating data and software that had been previously used in building resources used in the preparation of Bergey's Manual of Systematic Bacteriology, 2 nd Edition (Volumes 1 & 2A-C) and the Ribosomal Database Project-II (RDP-II) so as to both enhance the value of the data and create a pipeline approach to keeping the data current. Earlier, we demonstrated the value of using exploratory data analysis (EDA) to visualize large sets of sequence data (notably SSU rRNA gene sequences used in constructing a comprehensive phylogeny of prokaryotes. While the Self- Organizing Self-Correcting Classification (SOSCC) algorithms we developed were computationally efficient and useful for unraveling problems within the underlying data (e.g., identification of annotation errors, detection of unresolved synonymies, taxonomic and nomenclatural errors), bottlenecks at the preprocessing stage limited deployment of our applications as tools for end-users. To overcome the bottlenecks (which included hand alignment and computation of large matrices of pair-wise evolutionary distances), we proposed building a data pipeline between the "Taxomatic" application and RDP-II. The objectives were to accelerate the production and distribution of the updated versions of the prokaryotic taxonomy in lock-step with publication of new taxa and rearrangement of existing taxa, and to distribute these data more readily with the RDP-II and and other stakeholders in the community. A related goal of the current project is to deploy our visualization techniques as an interactive web application by which end-users can view manipulate, and select datasets of particular interest based upon phylogenetic and genomic information, access sequence data, and ultimately the scientific literature where the original observations were made and those that build on the original observations. The Taxomatic is a web-based tool to visualize distance matrices. The tool accepts raw distance matrices or aligned sequence information as data sources. When sequence information is provided the distance matrix is computed using the uncorrected distance model. Users can upload files to the Taxomatic website or sequences can be submitted by a SOAP service. This SOAP service is used by RDP to streamline Taxomatic use with RDP data. In addition to supplying source information, users can either supply their own taxonomic information by uploading it in XML, retrieve data taxonomic information from the RDP using either RDP or Genbank identifiers as source data, with or without classification by the RDP Classifier web service, or completely omit taxonomic data. In the latter case, the input distance matrix can be viewed in the order in which it was loaded.

59 BASIC BIOLOGICAL SCIENCES↗

Performance Analysis and Optimization for Scientific Data Workloads

Scientific data generated at experimental and observational facilities are increasingly being processed on large-scale compute systems. Most of the experimental data analysis workflows are not designed or implemented to run on large scale environments and take full advantage of HPC compute and storage resources. These applications are unlike the traditional tightly-coupled scientific applications and hence face significant performance and scalability challenges as the volume of data increases exponentially. In this paper, we conduct a performance and scalability analysis for experimental analysis applications and workflows operating on data from light sources. Our analysis detects and quantifies I/O performance, scalability and runtime bottlenecks for three data analysis applications that run on NERSC resources. Based on our analysis we propose and implement a set of optimizations that lead to reducing the amount of time spent on I/O operations by almost 90%.

97 MATHEMATICS AND COMPUTING↗

Hazard Detection Detector Cards

This report presents a comprehensive summary of five advanced anomaly detection tools developed and deployed by Oak Ridge National Laboratory in support of the VA’s Health Information Technology modernization. These detectors—Order Path Tracker, Trend Watcher, Pain Pointer, Performance Monitor, and Patient Record Flag Detector—leverage statistical and machine learning methods to monitor workflow disruptions, detect anomalies in care sequences and volumes, identify bottlenecks, and track system-level performance metrics across VistA and Millennium systems. All detectors have been integrated into the Health Data Analytics Platform (HDAP), with most having completed deployment and testing using live data from targeted stations in cardiology and oncology domains. This work enhances VA’s capacity for proactive system surveillance, promotes patient safety, and informs data-driven operational improvements across the EHR ecosystem.

97 MATHEMATICS AND COMPUTING↗

Preventing Failures By Dataset Shift Detection in Safety-Critical Graph Applications

Dataset shift refers to the problem where the input data distribution may change over time (e.g., between training and test stages). Since this can be a critical bottleneck in several safety-critical applications such as healthcare, drug-discovery, etc., dataset shift detection has become an important research issue in machine learning. Though several existing efforts have focused on image/video data, applications with graph-structured data have not received sufficient attention. Therefore, in this paper, we investigate the problem of detecting shifts in graph structured data through the lens of statistical hypothesis testing. Specifically, we propose a practical two-sample test based approach for shift detection in large-scale graph structured data. Our approach is very flexible in that it is suitable for both undirected and directed graphs, and eliminates the need for equal sample sizes. Using empirical studies, we demonstrate the effectiveness of the proposed test in detecting dataset shifts. We also corroborate these findings using real-world datasets, characterized by directed graphs and a large number of nodes.

97 MATHEMATICS AND COMPUTING↗