Engineering topics

Kiran, Mariam

Publications and source records attributed to Kiran, Mariam.

29 records · Page 2

DOE OSTI2020

End-to-end online performance data capture and analysis for scientific workflows

With the increased prevalence of employing workflows for scientific computing and a push towards exascale computing, it has become paramount that we are able to analyze characteristics of scientific applications to better understand their impact on the underlying infrastructure and vice-versa. Such analysis can help drive the design, development, and optimization of these next generation systems and solutions. Here, we present the architecture, integrated with existing well-established and newly developed tools, to collect online performance statistics of workflow executions from various, heterogeneous sources and publish them in a distributed database (Elasticsearch). Using this architecture, we are able to correlate online workflow performance data, with data from the underlying infrastructure, and present them in a useful and intuitive way via an online dashboard. We have validated our approach by executing two classes of real-world workflows, both under normal and anomalous conditions. The first is an I/O-intensive genome analysis workflow; the second, a CPU- and memory-intensive material science workflow. Based on the data collected in Elasticsearch, we are able to demonstrate that we can correctly identify anomalies that we injected. The resulting end-to-end data collection of workflow performance data is an important resource of training data for automated machine learning analysis.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Deep Reinforcement Learning in Buildings: Implicit Assumptions and their Impact

As deep reinforcement learning (DRL) continues to gain interest in the smart building research community, there is a transition from simulation-based evaluations to deploying DRL control strategies in actual buildings. While the efficacy of a solution could depend on a particular implementation, there are common obstacles that developers have to overcome to deliver an effective controller. Additionally, a deployment in a physical building can invalidate some of the assumptions made during the controller development. Assumptions on the sensor placement or on the equipment behavior can quickly come undone. This paper presents some of the significant assumptions made during the development of DRL based controllers that could affect their operations in a physical building. Furthermore, a preliminary evaluation revealed that controllers developed with some of these assumptions can incur twice the expected costs when they are deployed in a building.

Prakash, Anand Krishnan↗

DOE OSTI2020

Hylia: Time-Series Library for Network Operations (Hylia) v1

The ability to analyze time-series data in network operations is critical to ensure optimal operations and management in wide area networks, sensor networks, and cloud networks. Hylia is a collection of multiple time-series prediction algorithms designed specifically to work with network monitoring data sets such as SNMP, NETFLOW and other monitoring data. It is designed to recognize these features and perform the extrapolation of the data.

Kiran, Mariam↗

DOE OSTI2020

Indira: intent-based networking and rendering for networks (Indira) v1

Intelligent Network Deployment Intent Renderer Application an interactive network assistant that will help us configure a data path between two scientific end-point groups (EPGs) to optimize the transfer of elephant data flows. This tool is able to talk to scientists gather network requirements and then convert to network programming code and automate the end-to-end provisioning.

Pouyoul, Eric↗

DOE OSTI2020

Merope: Unsupervised Learning for Network Operations (Merope) v1

The ability to analyze network flow characteristics is critical to understand network patterns that can help us understand how our networks are behaving. Merope is a collection of many unsupervised learning approaches and visualizers to allow networks to be more visual and analytical.

Kiran, Mariam↗

DOE OSTI2020

Analysis of curtailment at The Geysers geothermal Field, California

Geothermal energy has traditionally been viewed as a baseload energy source, but the rapid growth of intermittent renewable energy has led to a need for more flexibility in power generation to avoid mandatory curtailment imposed by grid operators. This study of curtailment at The Geysers provides insights into the magnitude, duration, frequency, temporal and spatial distribution, and potential causes of curtailment events between 2013 and 2018. Annual levels of curtailment range during this period from 9 to 47 GW h, representing 0.15 to 0.81 % of the net generation. Most curtailments occurred at the power plants connected to a lower capacity transmission line and may result from transmission constriction. There is a clear link between negative pricing and economic curtailment, especially when solar production is higher. Economic curtailment events tend to be only a few hours and vary in magnitude up to almost 300 MW, whereas transmission-related curtailment events can be up to several weeks in duration. It is likely that curtailment of geothermal power will be an increasing concern, and could be mitigated by flexible generation strategies and increases in energy storage. It is critical to know the nature of curtailment events so that flexible generation options can be assessed properly.

15 GEOTHERMAL ENERGY↗

DOE OSTI2020

Detecting anomalous packets in network transfers: investigations using PCA, autoencoder and isolation forest in TCP

Large-scale scientific workflows rely heavily on high-performance file transfers. These transfers require strict quality parameters such as guaranteed bandwidth, no packet loss or data duplication. To have successful file transfers, methods such as predetermined thresholds and statistical analysis need to be done to determine abnormal patterns. Network administrators routinely monitor and analyze network data for diagnosing and alleviating these, making decisions based on their experience. However, as networks grow and become complex, monitoring large data files and quickly processing them, makes it improbable to identify errors and rectify these. Abnormal file transfers have been classified by simply setting alert thresholds, via tools such as PerfSonar and TCP statistics (Tstat). This paper investigates the feasibility of unsupervised feature extraction methods for identifying network anomaly patterns with three unsupervised classification methods—principal component analysis, autoencoder and isolation forest. Here, we collect file transfer statistics from two experiment sets—synthetic iPerf generated traffic and 1000 Genome workflow runs, with synthetically introduced anomalies. Our results show that while PCA and a simple autoencoder finds it difficult to detect clusters, the tree-variant isolation forest is able to identify anomalous packets by breaking down TCP traces into tree classes early.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

5G Enabled Energy Innovation: Advanced Wireless Networks for Science (Workshop Report)

Rapidly expanding, new telecommunications infrastructure based on 5G technologies will disrupt and transform how we design, build, operate, and optimize scientific infrastructure and the experiments and services enabled by that infrastructure, from continental-scale sensor networks to centralized scientific user facilities, from intelligent Internet of Things devices to supercomputers. Concurrently, 5G will introduce, or exacerbate, challenges related to protecting infrastructure and associated scientific data as well as to fully leveraging opportunities related to expanded infrastructure scale and complexity. The U.S. Department of Energy (DOE) Office of Science operates scientific infrastructure, supporting some of the nation’s most advanced intellectual discoveries, spanning the country and including 30 world-class user facilities from supercomputers to accelerators. Along with field experiments and remote observatories, every aspect of DOE’s scientific enterprise will be affected by 5G, which amounts to a complete renovation of the underpinnings of the nation’s information infrastructure. In this report we explore the scientific opportunities and new research challenges associated with 5G, ranging from scalability to heterogeneity to cybersecurity. The rapid commercial deployment of 5G opens the opportunity to rethink and reinvent DOE’s scientific infrastructure and experimentation, from intelligent sensor networks at unprecedented scales to a digital continuum of cyberinfrastructure spanning low-power sensors, high-performance computing embedded within and at the edge of the network, and DOE’s large-scale user instrument and computing facilities. New programming paradigms, workflow and data frameworks, and AI-based system design, operation, and autonomous adaptation and optimization will be necessary in order to exploit these new opportunities. Field deployments and centralized scientific instruments can also be revolutionized, moving (without traditional performance penalties) from wired to wireless connectivity for data and control systems, improving flexibility, and opening new sensing modalities, including the use of the 5G electromagnetic spectrum itself as an environmental probe. For DOE science, in contrast to commercial 5G applications and settings, devices will be deployed in extreme environments such as cryogenically cooled instrument control systems and in remote settings with harsh conditions, requiring the design of new materials for RF communication and edge processing to operate in these regimes. Concurrently, 5G infrastructure comprises both hardware and sophisticated software systems - currently closed and proprietary. The cybersecurity challenges to 5G-empowered reinvention mirror the complexity and variety of new 5G features, from virtualization to private network slices to ubiquitous access. Research is also needed in order to accelerate the development of secure and open 5G software infrastructure, reducing reliance on hardware and software produced outside the United States and providing the transparency and rigorous evaluation and testing afforded through open software. Twelve broad research thrusts are laid out in four chapters, with a companion fifth chapter (and three additional research thrusts) underscoring the needs and opportunities for an aggressive testbed program co-designed by networking experts and scientists involved in the 15 research thrusts. The urgency of undertaking this research is fueled by a global, accelerating deployment of new telecommunications infrastructure that is designed for entertainment and commercial applications - barely scratching the surface of what 5G can do to extend U.S. leadership in scientific discovery.

42 ENGINEERING↗

DOE OSTI2020

Do optimization methods in deep learning applications matter?

With advances in deep learning, exponential data growth and increasing model complexity, developing efficient optimization methods are attracting much research attention. Several implementations favor the use of Conjugate Gradient (CG) and Stochastic Gradient Descent (SGD) as being practical and elegant solutions to achieve quick convergence, however, these optimization processes also present many limitations in learning across deep learning applications. Recent research is exploring higher-order optimization functions as better approaches, but these present very complex computational challenges for practical use. Comparing first and higher-order optimization functions, in this paper, our experiments reveal that Levemberg-Marquardt (LM) significantly supersedes optimal convergence but suffers from very large processing time increasing the training complexity of both, classification and reinforcement learning problems. Our experiments compare off-the-shelf optimization functions(CG, SGD, LM and L-BFGS) in standard CIFAR, MNIST, CartPole and FlappyBird experiments.The paper presents arguments on which optimization functions to use and further, which functions would benefit from parallelization efforts to improve pretraining time and learning rate convergence.

Ozyildirim, Buse Melis↗

DOE OSTI2019

State-based network similarity visualization

We introduce an approach for the interactive visual analysis of weighted, dynamic networks. These networks arise in areas such as computational neuroscience, sociology, and biology. Network analysis remains challenging due to complex time-varying network behavior. For example, edges disappear/reappear, communities grow/vanish, or overall network topology changes. Our technique, TimeSum, detects the important topological changes in graph data to abstract the dynamic network and visualize one summary representation for each temporal phase, a state. We define a network state as a graph with similar topology over a specific time interval. To enable a holistic comparison of networks, we use a difference network to depict edge and community changes. Here, we present case studies to demonstrate that our methods are effective and useful for extracting and exploring complex dynamic behavior of networks.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2019

[Office of Basic Energy Sciences (BES)] Roundtable on Producing and Managing Large Scientific Data with Artificial Intelligence and Machine Learning

To identify specific Priority Research Opportunities (PROs) for artificial intelligence and machine learning (AI/ML) at the US Department of Energy’s scientific user facilities, the Office of Basic Energy Sciences (BES) convened a roundtable of facility experts encompassing the fields of physics, chemistry, materials synthesis science, computational science, detector and accelerator technology, theory, modeling, simulation, and atomic-scale characterization techniques. The roundtable met on October 22–23, 2019, to identify coordinated, long-term AI/ML research efforts that will drive major advances in neutron, photon, and nanoscale sciences. This report describes the four PROs identified at the roundtable: PRO 1 on how AI/ML can extract high-value information from the large datasets; PRO 2 on how AI/ML can use such information in real time to maximize the facilities’ scientific output; PRO 3 on using AI/ML virtual laboratories (i.e., computational models of experimental facilities) to aid the facilities and user community in design and control of machine parameters and the design and execution of experiments, including training AI/ML models for PROs 1 and 2; and PRO 4 on how a shared scientific data infrastructure can provide tools to assemble and analyze the totality of data coming from user facilities. A section of the end of the report provides a summary on computer science and mathematics that highlights where enhanced AI/ML capabilities could be particularly impactful for BES user facilities.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗