Search NASA⌕ Search

SEARCH · Search NASA

Results for “analysis workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29

Performance Visualization for TAU Instrumented Scientific Workflows

In exascale scientific computing, it is essential to efficiently monitor, evaluate and improve performance. Visualization and especially visual analytics are useful and inevitable techniques in the exascale computing era to enable such a human-centered experience. In this ongoing work, we present a visual analytics framework for performance evaluation of scientific workflows. Ultimately, we aim to solve two current challenges: the capability to deal with workflows, and the scalability toward exascale scenario. On the way to achieve these goals, in this work, we first incorporate TAU (Tuning and Analysis Utilities) instrumentation tool and improve it to accommodate workflow measurements. Then we establish a web-based visualization framework, whose back end handles data storage, query and aggregation, while front end presents the visualization and takes user interaction. In order to support the scalability, a few level-of-detail mechanisms are developed. Finally, a chemistry workflow use case is adopted to verify our methods

97 MATHEMATICS AND COMPUTING↗

Porting Adaptive Ensemble Molecular Dynamics Workflows to the Summit Supercomputer

Molecular dynamics (MD) simulations must take very small (femtosecond) integration steps in simulation-time to avoid numerical errors. Efficient use of parallel programming models and accelerators in state-of-the art MD programs now is pushing Moore’s limit for time-per-MD step. As a result, directly simulating timescales beyond milliseconds will not be attainable directly, even at exascale. However, concepts from statistical physics can be used to combine many parallel simulations to provide information about longer timescales and to adequately sample the simulation space, while preserving details about the dynamics of the system. Implementing such an approach requires a workflow program that allows adaptable steering of task assignments based on extensive statistical analysis of intermediate results. Here we report the implementation of such an adaptable workflow program to drive simulations on the Summit IBM Power System AC922, a pre-exascale supercomputer at the Oak Ridge Leadership Computing Facility (OLCF). We compare to experiences on Titan, Summit’s predecessor, report the performance of the workflow and its components, and describe the porting process. We find that using a workflow program managed by a Mongo database can provide the fault tolerance, scalable performance, task dispatch rate, and reconfigurability required for robust and portable implementation of ensemble simulations such as are used in enhanced-sampling molecular dynamics. This type of workflow generator can also be used to provide adaptive steering of ensemble simulations for other applications in addition to MD.

Ossyra, John R.↗

Report of the IMFIT Advisory Committee from the Meeting of 9/16/08 at General Atomics

The Committee commends and thanks the presenters and GA for providing clear and informative presentations. We present our responses to 5 questions raised in the Charge, as well as additional comments that may be useful for the project. (Question 1) The IMFIT framework is based on PYTHON and CCA; can IMFIT meet its long-term goals with this choice of language for framework? Our first recommendation is to change the terminology; it appears that the project is not really utilizing CCA (Common Component Architecture). The project is really about using Python to organize workflow among components that are spawned as processes via pipes from Python. We suggest that in describing the project, the team drop the designation 'CCA' and simply state that the IMFIT framework is based on using Python to control workflow among components. Concerning the specific question: we understand the principal goal of the project to be the development of an easy-to-use tool that efficiently integrates different physics modules to support experimental data analysis and modeling. Yes, Python and the workflow-based framework are appropriate choices for meeting this objective. The Committee, in examining the goal, discussed whether the IMFIT project was duplicative of other projects; there are several others in the U.S. and elsewhere with integrated modeling goals. In particular in the U.S., SWIM, PTRANSP, and the proto-FSP's. However this project occupies a reasonably unique niche through its emphases on integrating experimental data, use of existing components, its near-term focus, and the extent to which is targeted at experimentalist users. (Question 2) Will IMFIT benefit from the additional use of an interoperability language for framework such as BABEL? No, IMFIT is not really doing inter-language communication. In particular this will be the case as long as inter-component communication is via files--and that's OK so long as you aren't doing lots of short steps with communication at each step. (Question 3) IMFIT GUI is based on the public PyGTK toolkit; will IMFIT benefit from the additional use of other public Python graphic toolkits such as wxPython that is a cross-platform wrapper, or PyQt, or a commercial package such as IDL? PyGTK seems adequate for now. The team needs to decide if it needs more capability. Avoid licensed software that interferes with portability and the open-source goal (and thus in particular avoid IDL). (Question 4) Will IMFIT benefit from using a XML-RPC as alternative to sending files thru sockets? IMFIT could benefit by adopting some kind of self-describing format, such as XML or HDF5, but data can still be sent over sockets. This is part of a broader question: 'Should IMFIT adopt some kind of standard for communicating data between components?' To do anything other than what is done now requires modification of individual components (or convertors/wrappers). But moving to a self-describing standard is a good idea, which should be considered. You don't need to impose a standard, but you could decide on one and move toward it gradually, as the architecture doesn't impose a standard. But whether you adopt a standard and the pace of moving toward it is up to you. (Question 5) Will IMFIT benefit from any other available computational tool? (a) IMFIT would benefit from fuller use of an already incorporated tool--python language in the task execution specification, to enable desired features like branching; (b) If there is a high degree of data hierarchy, IMFIT should explore HDF5 as well as NETCDF and compare; (c) IMFIT would benefit from moving toward common data standards; (d) It would be useful for the IMFIT team to monitor developments in related projects such as the proto-FSP's and PTRANSP to ascertain if there are potentially useful tools.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Decaf: Decoupled Dataflows for In Situ High-Performance Workflows

Decaf is a dataflow system for the parallel communication of coupled tasks in an HPC workflow. The dataflow can perform arbitrary data transformations ranging from simply forwarding data to complex data redistribution. Decaf does this by allowing the user to allocate resources and execute custom code in the dataflow. All communication through the dataflow is efficient parallel message passing over MPI. The runtime for calling tasks is entirely message-driven; Decaf executes a task when all messages for the task have been received. Such a messagedriven runtime allows cyclic task dependencies in the workflow graph, for example, to enact computational steering based on the result of downstream tasks. Decaf includes a simple Python API for describing the workflow graph. This allows Decaf to stand alone as a complete workflow system, but Decaf can also be used as the dataflow layer by one or more other workflow systems to form a heterogeneous task-based computing environment. In one experiment, we couple a molecular dynamics code with a visualization tool using the FlowVR and Damaris workflow systems and Decaf for the dataflow. In another experiment, we test the coupling of a cosmology code with Voronoi tessellation and density estimation codes using MPI for the simulation, the DIY programming model for the two analysis codes, and Decaf for the dataflow. Such workflows consisting of heterogeneous software infrastructures exist because components are developed separately with different programming models and runtimes, and this is the first time that such heterogeneous coupling of diverse components was demonstrated in situ on HPC systems.

97 MATHEMATICS AND COMPUTING↗

Exploring the potential of using L-Band InSAR for the mapping of flooded vegetation in tropical wetlands

Wetlands play a critical role in global water and carbon cycles, yet monitoring their water extent remains difficult, particularly beneath dense vegetation. SAR-based techniques such as backscatter thresholding are limited by complex scattering mechanisms, while fully polarimetric SAR (PolSAR) data capable of detecting doublebounce scattering remain scarce. To address these challenges, this study evaluates the potential of Interferometric SAR (InSAR) for mapping water surfaces beneath vegetation, termed flooded vegetation, using the Atrato floodplain in Colombia as a case study. We develop an automated workflow combining InSAR fringe detection with local phase homogeneity analysis and random sampling of processing parameters to generate probabilistic flooded vegetation maps. Applied to ALOS PALSAR-1 L-band image pairs from 2007–2011, the workflow captures seasonal fluctuations in flooded extent ranging from 500 to 1,500 km2. Compared to other L-band SAR inundation products, the InSAR-based maps identify broader flooded areas, with ~70% agreement in pairwise comparisons. Around 84% of detections align with existing wetland inventories and seasonal changes correspond with regional hydrological indicators, including terrestrial water storage anomalies and water gauge measurements. PolSAR analysis shows that InSAR complements backscatter-based methods by detecting inundation in areas with weak double-bounce signals. These findings suggest that combining InSAR with backscatter-based methods can improve detection of flooded vegetation, which is especially relevant for the upcoming NISAR mission that will offer frequent global L-band observations.

Coastal inundation↗

Multi-scale Quantitative Risk Analysis of Seabed Minerals: Principles and Application to Seafloor Massive Sulfide Prospects

The potential for mining hydrothermal mineral deposits on the seafloor, such as seafloor massive sulfides, has become technically possible, and some companies (currently not many) are considering their exploration and development. Yet, no present methodology has been designed to quantify the ore potential and assess the risks relative to prospectivity at prospect and regional scales. Multi-scale exploration techniques, similar to those of the play analysis that are used in the oil and gas industry, can help to fulfill this task by identifying the characteristics of geologic environments indicative of ore-forming processes. Such characteristics can represent a combination of, e.g., heat source, pathway, trap and reservoir that all dictate how and where ore components are mobilized from source to deposition. In this study, the understanding of these key elements is developed as a mineral system, which serves as a guide for mapping the risk of the presence or absence of ore-forming processes within the region of interest (the permissive tract). The risk analysis is carried out using geoscience data, and it is paired with quantitative resource estimation analysis to estimate the in-place mineral potential. Resource estimates are simulated stochastically with the help of available data (bathymetric features in this study), conventional grade–tonnage models and Monte Carlo simulation techniques. In this paper, the workflow for a multi-scale quantitative risk analysis, from the definition to the evaluation of a permissive tract and related prospect(s), is described with the help of multi-beam data of a known hydrothermal vent site.

58 GEOSCIENCES↗

The PATRIC Bioinformatics Resource Center: expanding data and analysis capabilities

The PathoSystems Resource Integration Center (PATRIC) is the bacterial Bioinformatics Resource Center funded by the National Institute of Allergy and Infectious Diseases (https://www.patricbrc.org). PATRIC supports bioinformatic analyses of all bacteria with a special emphasis on pathogens, offering a rich comparative analysis environment that provides users with access to over 250 000 uniformly annotated and publicly available genomes with curated metadata. PATRIC offers web-based visualization and comparative analysis tools, a private workspace in which users can analyze their own data in the context of the public collections, services that streamline complex bioinformatic workflows and command-line tools for bulk data analysis. Over the past several years, as genomic and other omics-related experiments have become more cost-effective and widespread, we have observed considerable growth in the usage of and demand for easy-to-use, publicly available bioinformatic tools and services. Here we report the recent updates to the PATRIC resource, including new web-based comparative analysis tools, eight new services and the release of a command-line interface to access, query and analyze data.

59 BASIC BIOLOGICAL SCIENCES↗

Unraveling the impact of initial choices and in-loop interventions on learning dynamics in autonomous scanning probe microscopy

The current focus in Autonomous Experimentation (AE) is on developing robust workflows to conduct the AE effectively. This entails the need for well-defined approaches to guide the AE process, including strategies for hyperparameter tuning and high-level human interventions within the workflow loop. This paper presents a comprehensive analysis of the influence of initial experimental conditions and in-loop interventions on the learning dynamics of Deep Kernel Learning (DKL) within the realm of AE in scanning probe microscopy. We explore the concept of the “seed effect,” where the initial experiment setup has a substantial impact on the subsequent learning trajectory. Additionally, we introduce an approach of the seed point interventions in AE allowing the operator to influence the exploration process. Using a dataset from Piezoresponse Force Microscopy on PbTiO 3 thin films, we illustrate the impact of the “seed effect” and in-loop seed interventions on the effectiveness of DKL in predicting material properties. The study highlights the importance of initial choices and adaptive interventions in optimizing learning rates and enhancing the efficiency of automated material characterization. This work offers valuable insights into designing more robust and effective AE workflows in microscopy with potential applications across various characterization techniques.

47 OTHER INSTRUMENTATION↗

Nova Analysis (Final Technical Report)

As the adoption of solar plus storage technology is rapidly increasing, there is a need for a more wholistic view of homes adopting them. In homes, both energy efficiency (EE) upgrades and DERs provide value not just to the homeowner, but to utilities and even society at large through emissions. Traditionally, EE and DERs are separate sectors, which makes it difficult to understand the co-benefits of their adoption. To address this, we created a novel workflow of tools that allows for a complete analysis of both efficiency and DERs. We also looked at a suite of metrics designed to capture the different benefits provided to different stakeholders, demonstrated with multiple sets of field data. A final report demonstrates the potential of the simulation workflow by simulating hundreds of buildings spread across the U.S. and demonstrating how a variety of factors affect the optimal sizing of DERs along with several other key metrics including energy, utility bills, emissions, and average resilience hours.

14 SOLAR ENERGY↗

Enabling Re-executable Workflows with Near-realtime Visualization, Provenance Capture and Advanced Querying for Mass Spectrometry Data

Mass spectrometry imaging (MSI) enables simultaneous spatially-resolved analysis of numerous ionizable molecules on a sample surface. The current analysis pipeline involves analyzing the data coming off the instrument using a tool called MSI Quickview and saving the results onto a storage drive before moving on to the next experiment. While this works well for single datasets, there is a demand for more scalable, flexible workflows that are re-executable across datasets, support extensive querying and ease collaboration. Here, we present a workflow that moves data analysis from a mere desktop application for single experiments to a more general capability

Re-executable workflow↗

Optimizing mixed cool thermal storage systems across a connected community

A high level of electric demand flexibility must be integrated into our building infrastructure to enable greater renewable energy penetration in the grid. In the U.S., 9% of electricity generated is used to cool buildings in a periodic manner, making this end-use an ideal target for active management through cool thermal energy storage (CTES) technologies. Historic uses for CTES are designed around central chilled water plants, but these systems cool less than 25% of U.S. commercial floorspace. Emerging technologies are under development to serve the many smaller distributed cooling systems, such as rooftop units (RTUs), and have the potential to add CTES to an additional 66% of cooled commercial floorspace. However, these unitary thermal storage systems (UTSS) lack the modeling and analysis tools to evaluate them in the future interactive grid context. Here, this study develops the modeling and optimization tools necessary to simultaneously examine central and distributed ice storage systems within the multi-building, connected community context. An integrated simulation-optimization workflow is created to allow for rapid customized analysis. Results demonstrate the energy and flexibility tradeoffs of various implementations.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

MS Based Metabonomics

Metabonomics is the latest and least mature of the systems biology triad, which also includes genomics and proteomics, and has its origins in the early orthomolecular medicine work pioneered by Linus Pauling and Arthur Robinson. It was defined by Nicholson and colleagues in 1999 as the quantitative measurement of perturbations in the metabolite complement of an integrated biological system in response to internal or external stimuli, and is often used today to describe many non-global types of metabolite analyses. Applications of metabonomics are extensive and include toxicology, nutrition, pharmaceutical research and development, physiological monitoring and disease diagnosis. For example, blood samples from millions of neonates are tested routinely by mass spectrometry (MS) as a diagnostic tool for inborn errors of metabolism. The metabonome encompasses a wide range of structurally diverse metabolites; therefore, no single analytical platform will be sufficient. Specialized sample preparation and detection techniques are required, and advances in NMR and MS technologies have led to enhanced metabonome coverage, which in turn demands improved data analysis approaches. The role of MS in metabonomics is still evolving as instrumentation and software becomes more sophisticated and as researchers realize the strengths and limitations of current technology. MS offers a wide dynamic range, high sensitivity, and reproducible, quantitative analysis. These attributes are essential for addressing the challenges of metabonomics, as the range of metabolite concentrations easily exceeds nine orders of magnitude in biofluids, and the diversity of molecular species ranges from simple amino and organic acids to lipids and complex carbohydrates. Additional challenges arise in generating a comprehensive metabolite profile, downstream data processing and analysis, and structural characterization of important metabolites. A typical workflow of MS-based metabonomics is shown in Figure 1. Gas chromatography-(GC)-MS was the most commonly used MS-based method for small molecule analysis in the 1970s and 1980s. It is still used today for the detection of many metabolic disorders and plays a strong role in plant metabonomics. Liquid chromatography (LC)-MS approaches have grown in popularity for metabolite studies, due to simpler sample preparation, reduced analysis times through the introduction of ultra-high performance liquid chromatography (UPLC)-MS and the ability to observe a wider range of metabolites. This chapter will discuss the role of MS in metabonomics, the techniques involved in this exciting area, and the current and future applications of the field. The various bioinformatics tools and multivariate analysis techniques used to maximize information recovery and to aid in the interpretation of the very large data sets typically obtained in metabonomics studies will also be discussed. While there are many different MS-based approaches utilized in metabonomics studies, emphasis will be placed on more established methods.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Exploring the Optimal Platform Configuration for Power-Constrained HPC Workflows

In high-performance computing (HPC) workflows, data analytics is typically utilized to gain insights from scientific simulations. Approaching the era of exascale, online analysis is gaining popularity due to the savings of I/O to persistent storage. As computing capability keeps growing, power consumption is becoming critical to HPC facilities. Enforcing power limits is emerging as a practical trend for power-constrained HPC facilities. However, it remains unclear how to choose the appropriate power limits for various HPC workflows and how to distribute the power limit of a workflow between simulation and analysis. In addition, given a power limit, it is unclear what the optimal scales and power capping levels are for various workflows, especially when taking reliability into account. In order to resolve these issues in power-constrained HPC, in this paper, we propose a reliability-aware model to determine the aforementioned platform configurations for HPC workflows. We also validate our model and present model-driven studies for a wide range of real-system scenarios. Our study reveals interesting insights about how platform configuration affects the performance and energy efficiency of HPC workflows under power constraints.

Tang, Kun↗

Performance Analysis Tool for HPC and Big Data Applications on Scientific Clusters

Big data is prevalent in HPC computing. Many HPC projects rely on complex workflows to analyze terabytes or petabytes of data. These workflows often require running over thousands of CPU cores and performing simultaneous data accesses, data movements, and computation. It is challenging to analyze the performance involving terabytes or petabytes of workflow data or measurement data of the executions, from complex workflows over a large number of nodes and multiple parallel task executions. To help identify performance bottlenecks or debug the performance issues in large-scale scientific applications and scientific clusters, we have developed a performance analysis framework, using state-ofthe- art open-source big data processing tools. Our tool can ingest system logs and application performance measurements to extract key performance features, and apply the most sophisticated statistical tools and data mining methods on the performance data. It utilizes an efficient data processing engine to allow users to interactively analyze a large amount of different types of logs and measurements. To illustrate the functionality of the big data analysis framework, we conduct case studies on the workflows from an astronomy project known as the Palomar Transient Factory (PTF) and the job logs from the genome analysis scientific cluster. Our study processed many terabytes of system logs and application performance measurements collected on the HPC systems at NERSC. The implementation of our tool is generic enough to be used for analyzing the performance of other HPC systems and Big Data workows.

97 MATHEMATICS AND COMPUTING↗

Numerical Validation of an Algorithm for Combined Soiling and Degradation Analysis of Photovoltaic Systems

We describe and demonstrate an open-source algorithm for simultaneously quantifying degradation and soiling of photovoltaic (PV) systems from energy-production time series data. The new analysis is based on year-on-year degradation rate analysis combined with stochastic rate and recovery soiling analysis. The algorithm is designed to fit into the workflow provided by RdTools, a Python module maintained by NREL and collaboratively developed with the community, which provides a framework and functions for degradation and loss-factor analysis of PV field data. We demonstrate the method on numerically simulated PV data sets and show that it reduces the root-mean-square error of the P50 degradation rate estimate when soiling is present.

14 SOLAR ENERGY↗

Lambda-PFLOTRAN 1.0: a workflow for incorporating organic matter chemistry informed by ultra high resolution mass spectrometry into biogeochemical modeling

Abstract. Organic matter (OM) composition plays a central role in microbial respiration of dissolved organic matter and subsequent biogeochemical reactions. Here, a direct connection of organic matter chemistry and thermodynamics to reactive transport simulators has been achieved through the newly developed Lambda-PFLOTRAN workflow tool that succinctly incorporates carbon chemistry data generated from Fourier transform ion cyclotron resonance mass spectrometry (FTICR-MS) into reaction networks to simulate organic matter degradation and the resulting biogeochemistry. Lambda-PFLOTRAN is a Python-based workflow, executed through a Jupyter notebook interface, that digests raw FTICR-MS data, develops a representative reaction network based on substrate-explicit thermodynamic modeling (also termed lambda modeling due to its key thermodynamic parameter λ used therein), and completes a biogeochemical simulation with the open source, reactive flow and transport code PFLOTRAN. The workflow consists of the following five steps: configuration, thermodynamic (lambda) analysis, sensitivity analysis, parameter estimation, and simulation output and visualization. Two test cases are provided to demonstrate the functionality of the Lambda-PFLOTRAN workflow. The first test case uses laboratory incubation data of temporal oxygen depletion to fit lambda parameters (i.e., maximum utilization rate and microbial carrying capacity). A slightly more complex second test case fits multiple lambda formulation and soil organic matter release parameters to temporal greenhouse gas generation measured during a soil incubation. Overall, the Lambda-PFLOTRAN workflow facilitates upscaling by using molecular-scale characterization to inform biogeochemical processes occurring at larger scales.

58 GEOSCIENCES↗

AToM: Advanced Tokamak Modeling Environment

Stability constraints play an important role in integrated modeling. The global plasma stability and macro-instabilities such as internal kink modes, neoclassical tearing modes, and edge localized perturbations limit the plasma performance and can result in large-scale transient events and plasma disruptions. These macro-instabilities can be also beneficial to the plasma performance. For example, the peeling models often lead to the edge localized modes (ELMs). However, the formation of stationary edge harmonic oscillations (EHOs) due to nonlinear interaction of peeling modes can result in a transition to the Quiescent H-mode (QH-mode) without ELMs. The profiles predicted with transport models need to be refined using the MHD stability constraints. This is important especially for transient and nonlinear stages of discharges such as ramp-up access to hybrid and steady state operation, and L- to H-mode transition as well as transition to the QH-mode. Several stability and MHD codes are already included in the OMFIT framework. These codes include BALOO, ELITE, GATO, M3DC1, MARS, and NIMROD. However, the verifications of stability constraints are currently mostly excluded from transport modeling workflows. The only exception is the EPED module which includes the ELITE predictions to limit the pedestal height. Here, we improved the OMFIT workflow to include stability calculations. Including the stability conditions to the integrated modeling workflow improved the robustness of the predictive modeling discharges. Being implemented in the workflow, the stability conditions can be also used for the stability analysis of experimental data. This improved the physics understanding of various discharge scenarios and can be used in experiment planning.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

PANORAMA: An approach to performance modeling and diagnosis of extreme-scale workflows

Here we report that computational science is well established as the third pillar of scientific discovery and is on par with experimentation and theory. However, as we move closer toward the ability to execute exascale calculations and process the ensuing extreme-scale amounts of data produced by both experiments and computations alike, the complexity of managing the compute and data analysis tasks has grown beyond the capabilities of domain scientists. Therefore, workflow management systems are absolutely necessary to ensure current and future scientific discoveries. A key research question for these workflow management systems concerns the performance optimization of complex calculation and data analysis tasks. The central contribution of this article is a description of the PANORAMA approach for modeling and diagnosing the run-time performance of complex scientific workflows. This approach integrates extreme-scale systems testbed experimentation, structured analytical modeling, and parallel systems simulation into a comprehensive workflow framework called Pegasus for understanding and improving the overall performance of complex scientific workflows.

97 MATHEMATICS AND COMPUTING↗