Search NASA⌕ Search

Engineering topics

Altintas, Ilkay

Publications and source records attributed to Altintas, Ilkay.

At least 19 records

Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows

This report details recent progress for the ASCR funded project “Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows”. We refer to the project as IPPD/2, reflecting the 2017 renewal under expanded scope and partners In IPPD/2, we increased our research scope to include data motion. We are focusing on three major aspects: a) observe how data is generated, distributed, and used; b) analyze how data is (repeatedly) consumed with a focus both on repeated patterns and anomalies; and c) explore how to optimize data motion. This new work on data motion will augment and complement IPPD/2’s research that focused on the computational aspects of tasks. We leverage and extend our existing tools and demonstrate our work on the Belle II workflow suite as well as on workflows from NSLS-II. The highlights of our work are as follows: Provenance for Workflows: Provenance is used to provide information enabling quality control, re-run computational workflows, and reproduce results. IPPD/2 has been building a scalable provenance management system that enables the capture of provenance from the high-level workflow through all relevant system levels in one integrated environment. Leveraging this work, our recent efforts have included using provenance as an enabling technique. Workload characterization: Leveraging provenance and analysis, we characterize data movement within network, storage, and memory over a variety of workloads. This characterization enables an understanding by performance analysts and application developers of the range of behaviors that could be expected. Performance Prediction for Workflows: The goal of modeling distributed workflows is to understand performance bottlenecks and enable more intelligent task scheduling to optimize selected metrics of interest (e.g., task throughput or output data rate). IPPD/2 has utilized both analytical and AI/ML modeling methodologies for performance modeling. Advanced Scheduling and Fault Modeling for Workflows: Scheduling of large-scale scientific workflows on geographically distributed resources is a challenging problem. To improve workflow throughput, we combined novel scheduling algorithms with task predictions from performance modeling and fault modeling. Dynamically Alleviating Bottlenecks in Workflows: Exploiting our provenance, analysis, and modeling efforts, we have explored and developed several techniques for dynamically detecting and alleviating bottlenecks in data movement. In particular, we have spent considerable effort demonstrating our techniques on production-like workflow configurations.

97 MATHEMATICS AND COMPUTING↗

Workflows Community Summit 2022: A Roadmap Revolution

Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from the execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing (often referred to as a computing continuum) and the evolving needs of emerging scientific applications, it is paramount that the development of novel scientific workflows and system functionalities seek to increase the efficiency, resilience, and pervasiveness of existing systems and applications. Specifically, the proliferation of machine learning/artificial intelligence (ML/AI) workflows, need for processing large-scale datasets produced by instruments at the edge, intensification of near real-time data processing, support for long-term experiment campaigns, and emergence of quantum computing as an adjunct to HPC, have significantly changed the functional and operational requirements of workflow systems. Workflow systems now need to, for example, support data streams from the edge-to-cloud-to-HPC, enable the management of many small-sized files, allow data reduction while ensuring high accuracy, orchestrate distributed services (workflows, instruments, data movement, provenance, publication, etc.) across computing and user facilities, among others. Further, to accelerate science, it is also necessary that these systems implement specifications/standards and APIs for seamless (horizontal and vertical) integration between systems and applications, as well as enable the publication of workflows and their associated products according to the FAIR principles.

97 MATHEMATICS AND COMPUTING↗

Integrating plant physiology into simulation of fire behavior and effects

Summary Wildfires are a global crisis, but current fire models fail to capture vegetation response to changing climate. With drought and elevated temperature increasing the importance of vegetation dynamics to fire behavior, and the advent of next generation models capable of capturing increasingly complex physical processes, we provide a renewed focus on representation of woody vegetation in fire models. Currently, the most advanced representations of fire behavior and biophysical fire effects are found in distinct classes of fine‐scale models and do not capture variation in live fuel (i.e. living plant) properties. We demonstrate that plant water and carbon dynamics, which influence combustion and heat transfer into the plant and often dictate plant survival, provide the mechanistic linkage between fire behavior and effects. Our conceptual framework linking remotely sensed estimates of plant water and carbon to fine‐scale models of fire behavior and effects could be a critical first step toward improving the fidelity of the coarse scale models that are now relied upon for global fire forecasting. This process‐based approach will be essential to capturing the influence of physiological responses to drought and warming on live fuel conditions, strengthening the science needed to guide fire managers in an uncertain future.

54 ENVIRONMENTAL SCIENCES↗

How open data and interdisciplinary collaboration improve our understanding of space weather: A risk and resiliency perspective

Space weather refers to conditions around a star, like our Sun, and its interplanetary space that may affect space- and ground-based assets as well as human life. Space weather can manifest as many different phenomena, often simultaneously, and can create complex and sometimes dangerous conditions. The study of space weather is inherently trans-disciplinary, including subfields of solar, magnetospheric, ionospheric, and atmospheric research communities, but benefiting from collaborations with policymakers, industry, astrophysics, software engineering, and many more. Effective communication is required between scientists, the end-user community, and government organizations to ensure that we are prepared for any adverse space weather effects. With the rapid growth of the field in recent years, the upcoming Solar Cycle 25 maximum, and the evolution of research-ready technologies, we believe that space weather deserves a reexamination in terms of a “risk and resiliency” framework. By utilizing open data science, cross-disciplinary collaborations, information systems, and citizen science, we can forge stronger partnerships between science and industry and improve our readiness as a society to mitigate space weather impacts. The objective of this manuscript is to raise awareness of these concepts as we approach a solar maximum that coincides with an increasingly technology-dependent society, and introduce a unique way of approaching space weather through the lens of a risk and resiliency framework that can be used to further assess areas of improvement in the field.

79 ASTRONOMY AND ASTROPHYSICS↗

Smart connected worker edge platform for smart manufacturing: Part 1—Architecture and platform design

Abstract The challenge of sustainably producing goods and services for healthy living on a healthy planet requires simultaneous consideration of economic, societal, and environmental dimensions in manufacturing. Enabling technology for data driven manufacturing paradigms like Smart Manufacturing (a.k.a. Industry 4.0) serve as the technological backbone from which sustainable approaches to manufacturing can be implemented. Unfortunately, these technologies are typically associated with broader and deeper factory automation that is often too expensive and complex for the small and medium sized manufacturers (SMMs) that comprise the majority of manufacturing business in the USA and for whom their most valuable asset are the people whose jobs automation while replace. This paper describes an edge intelligent platform to integrate internet‐of‐things technologies with computing hardware, software, computational workflows for machine learning, and data ingestion, enabling SMMs to transition into smart manufacturing paradigms by leveraging the intelligence of their people. The platform leverages consumer grade electronics and sensors (affordable and portable), customized software with open source software packages (accessible), and existing communication network infrastructures (scalable). The software systems are implemented via Kubernetes orchestration of Docker containerization to ensure scalability and programmability. The platform is adaptive via computational workflow engines that produce information from data by processing with low‐cost edge computing devices while efficiently accessing resources of cloud servers as needed. The proposed edge platform connects workers to technological resources that provide computational intelligence (i.e., silicon‐based sensing and computation for data collection and contextualization) to enable decision making at the edge of advanced manufacturing.

Kim, Yoon G.↗

Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows (IPPD) (Final Report)

This report details the accomplishments from the ASCR funded project “Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows” under the award numbers FWP-66406 and DE-SC0012630, with a focus on the UC San Diego (Award No. DE-SC0012630) part of the accomplishments. We refer to the project as IPPD. The main activities of IPPD were centered on the development and integration of provenance information to capture empirically workflow information and identify the sources of bottlenecks as well as variability, and modeling and simulation to provide insights and predictively explore multiple scenarios for development of advanced techniques for optimization of resources and workflow execution. After a Phase I of the project, a Phase II effort focused on three major aspects: a) observe how data is generated, distributed, and used; b) analyze how data is (repeatedly) consumed with a focus both on repeated patterns and anomalies; and c) explore how to optimize data motion. The project leveraged and extended our existing tools with new research and demonstrated our work on the Belle II workflow suite as well as on workflows from NSLS-II.

97 MATHEMATICS AND COMPUTING↗

A Community Roadmap for Scientific Workflows Research and Development

The landscape of workflow systems for scientific applications is notoriously convoluted with hundreds of seemingly equivalent workflow systems, many isolated research claims, and a steep learning curve. To address some of these challenges and lay the groundwork for transforming workflows research and development, the WorkflowsRI and ExaWorks projects partnered to bring the international workflows community together. This paper reports on discussions and findings from two virtual “Workflows Community Summits” (January and April, 2021). The overarching goals of these workshops were to develop a view of the state of the art, identify crucial research challenges in the workflows community, articulate a vision for potential community efforts, and discuss technical approaches for realizing this vision. To this end, participants identified six broad themes: FAIR computational workflows; AI workflows; exascale challenges; APIs, interoperability, reuse, and standards; training and education; and building a workflows community. We summarize discussions and recommendations for each of these themes.

Ferreira Da Silva, Rafael↗

Modular performance prediction for scientific workflows using Machine Learning

Scientific workflows provide an opportunity for declarative computational experiment design in an intuitive and efficient way. A distributed workflow is typically executed on a variety of resources, and it uses a variety of computational algorithms or tools to achieve the desired outcomes. Such a variety imposes additional complexity in scheduling these workflows on large scale computers. As computation becomes more distributed, insights into expected workload that a workflow presents become critical for effective resource allocation. In this paper, we present a modular framework that leverages Machine Learning for creating precise performance predictions of a workflow. The central idea is to partition a workflow in such a way that makes the task of forecasting each atomic unit manageable and gives us a way to combine the individual predictions efficiently. We recognize a combination of an executable and a specific physical resource as a single module. This gives us a handle to characterize workload and machine power as a single unit of prediction. Overall, our modular technique of creating atomic modules and deployment of longest-path approach to estimate workflow performance, allows the framework to adapt to highly complex nested directed acyclic workflows and scale to new scenarios, since it does not make assumptions of underlying workflow structure. We present performance estimation results of independent workflow modules executed on the XSEDE SDSC Comet cluster using various Machine Learning algorithms. The results provide insights into the behavior and effectiveness of different algorithms in the context of scientific workflow performance prediction.

97 MATHEMATICS AND COMPUTING↗

Application of Deep Learning on Integrating Prediction, Provenance, and Optimization

In this research, we investigated two approaches to detect job anoma- lies and/or contention for large scale computing efforts: 1. Preemptive job scheduling using binomial classification long short-term memory networks 2. Forecasting intra-node computing loads from the active jobs and addi- tional job(s) For approach 1, we achieved a 14% improvement in computational resources utilization and an overall classification accuracy of 85% on real tasks executed in a High Energy Physics computing workflow. For this paper, we present the preliminary results used in second approach.

Anomaly Detection, Belle II Computing, data proven↗

Deep Learning for Enhancing Fault Tolerant Capabilities of Scientific Workflows

Deep Learning for Enhancing Fault TolerantCapabilities of Scientific WorkflowsAlok Singh, Ilkay AltintasSan Diego Supercomputer Center, UCSDLa Jolla, CA, USA{a1singh, ialtintas}@ucsd.eduMalachi Schram, Nathan TallentPacific Northwestern National LaboratoryRichland, WA, USA{malachi.schram, nathan.tallent}@pnnl.govAbstract-In the history of Computer Science, the act of ‘delegation’ has been the greatest multiplier of society’s problem-solving ability. A scientist working on detecting anomalies in a phenomenon, does not need to re-invent matrix multiplication techniques to solve her problem. Scientific workflows provide ultimate ‘delegation’ mechanism -where a domain scientist can completely forget the specifics of ‘how’ her program will execute on a large cluster in an efficient and cost-effective manner and can instead focus on the mathematical formulation and theoretical robustness of her solution. We present here an approach that directly aims to make the execution of Scientific Workflows more reliable, robust and efficient. We aim that the work presented in this paper will propel the larger effort, from the scientific workflow community, of making scientific workflow execution as simple, efficient and robust as a JOIN operation in a modern database. Specifically, we apply Deep Learning techniques to develop a mechanism that forecasts the final state (success or failure) of a dynamic job in a large-scale particle physics experiment, with minimal data gathering, and as early as possible in job’s life cycle. The key advantage of having a predictive mechanism to identify and anticipate failure-prone jobs is the potential for designing intelligent Fault Tolerance mechanisms to handle anomalous events. We achieve a 14% improvement in computational resources utilization, and an overall classification accuracy of 85% on real tasks executed in a High Energy Physics Computing workflow.To the best of our knowledge, this is the most exhaustive and first of its kind study of neural network architectures in context of a real-dataset profiled from a large-scalescientific workflow.

deep learning↗

Deep Learning on Operational Facility Data Related to Large-Scale Distributed Area Scientific Workflows

Abstract—Distributed computing platforms provide a robust mechanism to perform large-scale computations by splitting the task and data among multiple locations, possibly located thousands of miles apart geographically. This distribution of resources can lead to benefits such as redundancy of data, engagement with scientific teams that involve a diverse group of domain experts distributed spatially around the globe. It allows scientists to share their expertise and data with everyone and achieve insights by deploying collective intelligence of entire team. However, this distributed computing comes with its associated problems such as rampant duplication of file transfers increasing congestion, long job completion times, unexpected site crashing, suboptimal data transfer rates, unpredictable reliability in a time range, overshooting congestion, suboptimal usage of storage elements. In addition, each sub-system becomes a potential failure node that can trigger system wide disruptions.

Machine Learning, Provenance, Time Series, Semanti↗

The future of scientific workflows

Today’s computational, experimental, and observational sciences rely on computations that involve many related tasks. The success of a scientific mission often hinges on the computer automation of these workflows. In April 2015, the US Department of Energy (DOE) invited a diverse group of domain and computer scientists from national laboratories supported by the Office of Science, the National Nuclear Security Administration, from industry, and from academia to review the workflow requirements of DOE’s science and national security missions, to assess the current state of the art in science workflows, to understand the impact of emerging extreme-scale computing systems on those workflows, and to develop requirements for automated workflow management in future and existing environments. Here, this paper is a summary of the opinions of over 50 leading researchers attending this workshop. We highlight use cases, computing systems, workflow needs and conclude by summarizing the remaining challenges this community sees that inhibit large-scale scientific workflows from becoming a mainstream tool for extreme-scale science.

97 MATHEMATICS AND COMPUTING↗

Data Provenance Hybridization Supporting Extreme-Scale Scientific WorkflowApplications

As high performance computing (HPC) infrastructures continue to grow in capability and complexity, so do the applications that they serve. HPC and distributed-area computing (DAC) (e.g. grid and cloud) users are looking increasingly toward workflow solutions to orchestrate their complex application coupling, pre- and post-processing needs To gain insight and a more quantitative understanding of a workflow’s performance our method includes not only the capture of traditional provenance information, but also the capture and integration of system environment metrics helping to give context and explanation for a workflow’s execution. In this paper, we describe IPPD’s provenance management solution (ProvEn) and its hybrid data store combining both of these data provenance perspectives.

provenance↗

Enabling Structured Exploration of Workflow Performance Variability in Extreme-Scale Environments

Workflows are taking an Workflows are taking an increasingly important role in orchestrating complex scientific processes in extreme scale and highly heterogeneous environments. However, to date we cannot reliably predict, understand, and optimize workflow performance. Sources of performance variability and in particular the interdependencies of workflow design, execution environment and system architecture are not well understood. While there is a rich portfolio of tools for performance analysis, modeling and prediction for single applications in homogenous computing environments, these are not applicable to workflows, due to the number and heterogeneity of the involved workflow and system components and their strong interdependencies. In this paper, we investigate workflow performance goals and identify factors that could have a relevant impact. Based on our analysis, we propose a new workflow performance provenance ontology, the Open Provenance Model-based WorkFlow Performance Provenance, or OPM-WFPP, that will enable the empirical study of workflow performance characteristics and variability including complex source attribution.

Enabling structured workflows↗

Scientific Data Management (SDM) Center for Enabling Technologies. 2007-2012

Over the past five years, our activities have both established Kepler as a viable scientific workflow environment and demonstrated its value across multiple science applications. We have published numerous peer-reviewed papers on the technologies highlighted in this short paper and have given Kepler tutorials at SC06,SC07,SC08,and SciDAC 2007. Our outreach activities have allowed scientists to learn best practices and better utilize Kepler to address their individual workflow problems. Our contributions to advancing the state-of-the-art in scientific workflows have focused on the following areas. Progress in each of these areas is described in subsequent sections. Workflow development. The development of a deeper understanding of scientific workflows "in the wild" and of the requirements for support tools that allow easy construction of complex scientific workflows; Generic workflow components and templates. The development of generic actors (i.e.workflow components and processes) which can be broadly applied to scientific problems; Provenance collection and analysis. The design of a flexible provenance collection and analysis infrastructure within the workflow environment; and, Workflow reliability and fault tolerance. The improvement of the reliability and fault-tolerance of workflow environments.

97 MATHEMATICS AND COMPUTING↗

Scientific Data Management Integrated Software Infrastructure Center (SDM/ISIC): Scientific Process Automation (SPA) (Final Report)

The KEPLER project has made a significant impact and established a sustained leadership in the field of scientific workflows. The open source KEPLER project started in 2003 when members of the SciDAC-SDM Center/SPA Team, sponsored by this DOE-funded project (DE-FC02-01ER25486) and members of the SEEK project (NSF/ITR awards DBI-0225674 and DBI-0533368) decided to collaborate and jointly develop a scientific workflow system based on the open source PTOLEMY II system (Ludäscher was a co-PI on both projects). In the first project phase, between 2001 and 2003, the SDSC team (Altintas, Ludäscher) worked closely with a domain scientist (Matt Coleman, LLNL) and created early versions of a Promoter Identification Workflow (PIW). Towards the end of that period, the open source PTOLEMY II system was adopted by SciDAC-SDM and SEEK as the basis for a general scientific workflow system and problem-solving environment to design and execute scientific workflows, giving rise to Kepler. Once started through SciDAC-SDM and SEEK, the KEPLER leadership team was able to attract further support from funding agencies including from DOE (as part of SciDAC-2), NSF, NIH, and the Gordon and Betty Moore Foundation. Important advances to KEPLER were made by the SciDAC/SDM-SPA team during the report period (2001–2007), in particular by the SDSC team (and after Ludäscher's move) the UC Davis team, including (i) research and development of the workflow infrastructure, (ii) library (actor) development, and (iii) development of concrete scientific workflows, in collaboration with scientists. Years later, the KEPLER system was listed as one of the prominent “Big Data” research outcomes of DOE in the White House Big Data Factsheet, announced as part of the White House Big Data press release.

97 MATHEMATICS AND COMPUTING↗