Engineering topics

Kettimuthu, R.

Publications and source records attributed to Kettimuthu, R..

9 records

DOE OSTI2022

ROI-Finder : machine learning to guide region-of-interest scanning for X-ray fluorescence microscopy

The microscopy research at the Bionanoprobe (currently at beamline 9-ID and later 2-ID after APS-U) of Argonne National Laboratory focuses on applying synchrotron X-ray fluorescence (XRF) techniques to obtain trace elemental mappings of cryogenic biological samples to gain insights about their role in critical biological activities. The elemental mappings and the morphological aspects of the biological samples, in this instance, the bacterium Escherichia coli ( E. Coli ), also serve as label-free biological fingerprints to identify E. coli cells that have been treated differently. The key limitations of achieving good identification performance are the extraction of cells from raw XRF measurements via binary conversion, definition of features, noise floor and proportion of cells treated differently in the measurement. Automating cell extraction from raw XRF measurements across different types of chemical treatment and the implementation of machine-learning models to distinguish cells from the background and their differing treatments are described. Principal components are calculated from domain knowledge specific features and clustered to distinguish healthy and poisoned cells from the background without manual annotation. The cells are ranked via fuzzy clustering to recommend regions of interest for automated experimentation. The effects of dwell time and the amount of data required on the usability of the software are also discussed.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2019

Learning concave-convex profiles of data transport over dedicated connections

Dedicated data transport infrastructures are increasingly being deployed to support distributed big-data and high-performance computing scenarios. These infrastructures employ data transfer nodes that use sophisticated software stacks to support network transport among sites, which often house distributed file and storage systems. Throughput measurements collected over such infrastructures for a range of round trip times (RTTs) reflect the underlying complex end-to-end connections, and have revealed dichotomous throughput profiles as functions of RTT. In particular, concave regions of throughput profiles at lower RTTs indicate near-optimal performance, and convex regions at higher RTTs indicate bottlenecks due to factors such as buffer or credit limits. We present a machine learning method that explicitly infers these concave and convex regions and transitions between them using sigmoid functions. We also provide distribution-free confidence estimates for the generalization error of these concave-convex profile estimates. Throughput profiles for data transfers over 10 Gbps connections with 0–366 ms RTT provide important performance insights, including the near optimality of transfers performed with the XDD tool between XFS filesystems, and the performance limits of wide-area Lustre extensions using LNet routers. A direct application of generic machine learning packages does not adequately highlight these critical performance regions or provide as precise confidence estimates.

Rao, Nageswara↗

DOE OSTI2018

Democratizing Network Reservations through Application-Aware Orchestration

The provisioning of network connections for data transfers that provide quality of service (QoS) over research and education (R&E) networks is currently performed by network operators. For network connections that span multiple administrative domains, network operators have to reach agreements on reservation requirements. As a result, a network reservation request may take from days to weeks to be provisioned. To improve provisioning times and the success rate of multidomain network reservations, we designed and implemented an application-aware orchestration framework for multidomain R&E networks. This framework leverages latest developments in software-defined networking to automate network provisioning in order to democratize access to network reservation through novel APIs. We present the design, implementation, and evaluation of our application-aware orchestration framework. We evaluate our system using Mininet and demonstrate that it provisions 49% more reservations than current state-of-the-art systems within seconds of receiving a request.

Rao, Nageswara S.↗

DOE OSTI2018

Regression-Based Analytics for Response Dynamics of SDN Solutions and Components

We consider Software-Defined Network (SDN) solutions that dynamically setup and teardown connections, and also their response dynamics of connectivity and TCP throughput. There is a wide variety of choices for controllers, switches, andsoftware modules, and objective analytics of measurements are needed to compare the response dynamics of solutions based on their combinations. Throughput and ping measurements during connection changes contain significant statistical variations due to the complexities of controllers, northbound scripts, and network devices. We propose analytical methods to estimate response regression functions, which provide a rigorous and objective comparison of different SDN solutions and their components, namely, controllers, switches, and OpenFlow versions. We apply this approach in two scenarios: (a) switching of long-haul connections emulated in hardware by ANUE devices over a testbed consisting of HP and Cisco switches, and (b) multi-site science infrastructure emulated using VMs and Mininet. The results provide useful practical insights including: (i) the dpctl method responds seconds faster than the OpenDayLight (ODL) method on average under OpenFlow 1.0, but performs similar to ODL under OpenFlow 1.3, and (ii) the ping response of dpctl is about 0.6 sec faster than ODL for the second scenario.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

Virtual Environment for Testing Software-Defined Networking Solutions for Scientific Workflows

Recent developments in software-defined infrastructures promise that scientific workflows utilizing supercomputers, instruments, and storage systems will be dynamically composed and orchestrated using software at unprecedented speed and scale in the near future. Testing of the underlying networking software, particularly during initial exploratory stages, remains a challenge due to potential disruptions, and resource allocation and coordination needed over the multi-domain physical infrastructure. To overcome these challenges, we develop the Virtual Science Network Environment (VSNE) that emulates the multi-site host, storage, and network infrastructure using Virtual Machines (VMs), wherein the production and nascent software can be tested. Within each VM, which represents a site, the hosts and local-area networks are emulated using Mininet, and the Software-Defined Network (SDN) controllers and service daemon codes are natively run to support dynamic provisioning of network connections. Additionally, Lustre filesystem support at the sites and an emulation of the long-haul network using Mininet, are provided using separate VMs. As case studies, we describe Lustre file transfers using XDD, Red5 streaming service demonstration, and an emulated experiment with remote monitoring and steering modules, all supported over dynamically configured connections using SDN controllers.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

Cross-geography scientific data transferring trends and behavior

Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.

Liu, Zhengchun↗

DOE OSTI2018

So ware-Defined Network Solutions for Science Scenarios: Performance Testing Framework and Measurements

High-performance scientific work flows utilize supercomputers, scientific instruments, and large storage systems. Their executions require fast setup of a small number of dedicated network connections across the geographically distributed facility sites. We present Software-Defined Network (SDN) solutions consisting of site daemons that use dpctl, Floodlight, ONOS, or OpenDaylight controllers to set up these connections. The development of these SDN solutions could be quite disruptive to the infrastructure, while requiring a close coordination among multiple sites; in addition, the large number of possible controller and device combinations to investigate could make the infrastructure unavailable to regular users for extended periods of time. In response, we develop a Virtual Science Network Environment (VSNE) using virtual machines, Mininet, and custom scripts that support the development, testing, and evaluation of SDN solutions, without the constraints and expenses of multi-site physical infrastructures; furthermore, the chosen solutions can be directly transferred to production deployments. By complementing VSNE with a physical testbed, we conduct targeted performance tests of various SDN solutions to help choose the best candidates. In addition, we propose a switching response method to assess the setup times and throughput performances of different SDN solutions, and present experimental results that show their advantages and limitations.

Settlemyer, Bradley↗

DOE OSTI2017

Data Transfer Advisor with Transport Profiling Optimization

The network infrastructures have been rapidly upgraded in many high-performance networks (HPNs). However, such infrastructure investment has not led to corresponding performance improvement in big data transfer, especially at the application layer, largely due to the complexity of optimizing transport control on end hosts. We design and implement ProbData, a PRofiling Optimization Based DAta Transfer Advisor, to help users determine the most effective data transfer method with the most appropriate control parameter values to achieve the best data transfer performance. ProbData employs a profiling optimization based approach to exploit the optimal operational zone of various data transfer methods in support of big data transfer in extreme scale scientific applications. We present a theoretical framework of the optimized profiling approach employed in ProbData as wellas its detailed design and implementation. The advising procedure and performance benefits of ProbData are illustrated and evaluated by proof-of-concept experiments in real-life networks.

Rao, Nageswara S.↗

DOE OSTI2017

On Analytics of File Transfer Rates over Dedicated Wide-Area Connections

File transfers between the decentralized storage sites over dedicated wide-area connections are becoming increasingly important in high-performance computing and big data scenarios. Designing such scientific workflows for large file transfers is extremely challenging as they depend on the file, I/O, host, and local- and wide-area network subsystems, and their interactions. To gain insights into file-transfer rate profiles, we develop polynomial, bagging, and boosting regression models for Lustre and XFS file transfer measurements, which are collected using XDD over a suite of 10 Gbps connections with 0-366 ms round trip times (RTTs). In addition to overall trends and analytics, these regressions also provide file-transfer rate estimates for RTTs and number of parallel flows at which measurements might not have been collected. They show that bagging and boosting techniques provide closer data fits than the polynomial regression. We develop probabilistic bounds on the generalization error of these methods, which combined with the cross-validation error establish that former two are more accurate estimators than the polynomial regression. In addition, we present a method to efficiently determine the number of parallel flows to achieve a peak file-transfer rate using fewer than full sweep measurements; in our measurements, the peak is achieved in 96% of cases with 15-25% of measurements of a full sweep.

Sen, Satyabrata↗