Search NASA⌕ Search

SEARCH · Search NASA

Results for “analysis workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 811 records · Page 45

Python and HPC for High Energy Physics Data Analyses

High level abstractions in Python that can utilize computing hardware well seem to be an attractive option for writing data reduction and analysis tasks. In this paper, we explore the features available in Python which are useful and efficient for end user analysis in High Energy Physics (HEP). A typical vertical slice of an HEP data analysis is somewhat fragmented: the state of the reduction/analysis process must be saved at certain stages to allow for selective reprocessing of only parts of a generally time-consuming workflow. Also, algorithms tend to to be modular because of the heterogeneous nature of most detectors and the need to analyze different parts of the detector separately before combining the information. This fragmentation causes difficulties for interactive data analysis, and as data sets increase in size and complexity (O10 TiB for a “small” neutrino experiment to the O10 PiB currently held by the CMS experiment at the LHC), data analysis methods traditional to the field must evolve to make optimum use of emerging HPC technologies and platforms. Mainstream big data tools, while suggesting a direction in terms of what can be done if an entire data set can be available across a system and analysed with high-level programming abstractions, are not designed with either scientific computing generally, or modern HPC platform features in particular, such as data caching levels, in mind. Our example HPC use case is a search for a new elementary particle which might explain the phenomenon known as “Dark Matter”. Here, using data from the CMS detector, we will use HDF5 as our input data format, and MPI with Python to implement our use case.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Toward the Neutrino Discovery Platform: An Auditable, Uncertainty-Bearing Toolchain for MINERvA Open-Data Cross-Section Analysis

The Neutrino Discovery Platform (NDP) aims to accelerate DUNE-era science by making the neutrino program's existing datasets analyzable through fast, reproducible, and auditable workflows. We report a working version of two of its layers, data curation and agentic orchestration, built and tested end to end on MINERvA open data. The guiding lesson throughout is that a cross section is a measurement, and not just a plotted shape, only if it carries a defensible systematic-uncertainty budget, a trustworthy unfolding, and a reproducible record. Using a single medium-energy playlist pair from the MINERvA open-data release (about $2.05\times10^{17}$ protons on target of data), we first reproduced the shapes of two published charged-current inclusive $\nu_\mu$ measurements through a complete extraction ladder: selection, background subtraction, D'Agostini unfolding, efficiency correction, and flux normalization. These shape-level reproductions ran and tracked the published results, but they lacked the systematic-uncertainty machinery that defines a MINERvA cross section. To supply it, we vendored and built the MINERvA Analysis Toolkit and developed a many-universe systematic-uncertainty tool that produces a portable covariance artifact, a parallel event-loop runner, and a per-run auditability harness. Validated against a published covariance release, the toolchain reproduces the released statistical, flux, and muon-energy-scale terms and shows that they account for roughly 63\% of the total variance, with the remainder unreleased. Using this same infrastructure, we then performed a measurement of our own design, the hadronic recoil-energy distribution of low-energy ($E_\nu<2.5$~GeV) charged-current inclusive events, and found data/simulation shape agreement of $\chi^2/\mathrm{ndf}=1.26$. Together these results show that the platform supports original physics and not only reproductions.

Breaux, Auto [Tulane U. (main)]↗

Co-Firing Switchgrass and Waste Coal in A Power Plant: A Techno-Economic and Life Cycle Evaluation for The Ohio River Valley (SWITCH) (Final Technical Report for Ohio State/FE0032204)

Abandoned coal mine lands (AMLs) represent one of the most persistent environmental challenges in the United States. Prior to the enactment of the Surface Mining Control and Reclamation Act (SMCRA) in 1977, coal mining operations were not legally required to reclaim disturbed lands, leaving behind approximately 500,000 AML sites nationwide. These sites pose severe environmental and health risks, including acid mine drainage, soil and water contamination, and spontaneous combustion of waste coal piles. Millions of Americans live within one mile of these AMLs, underscoring the urgency of remediation. Traditional reclamation practices, such as planting cool-season grasses, often fail to fully restore ecological function or leverage the economic potential of these lands. This project addressed these challenges by developing integrated strategies for resource recovery, land reclamation, and sustainable energy production. This project evaluated an integrated strategy to convert this liability into an opportunity by recovering waste coal and co-firing it with switchgrass (Panicum virgatum L.) cultivated on reclaimed or marginal AML areas in existing coal-fired power plants. Switchgrass not only provides a renewable feedstock but also aids in land reclamation and carbon sequestration. 1) Remote Sensing and Machine Learning for Waste Coal Identification Using Sentinel-2 satellite imagery and supervised classification, we applied four machine learning models to detect historical waste coal piles. Random Forest achieved the highest accuracy (precision: 86%, recall: 77%). Time-series analysis revealed gradual vegetation recovery since 1986, indicating natural reclamation processes in historical sites, while active mining areas showed ongoing disturbance. This workflow enables scalable monitoring and prioritization of reclamation efforts. 2) UAS-Based Stockpile Volume Estimation To quantify recoverable waste coal, we evaluated Unmanned Aerial Systems (UAS) equipped with Light Detection and Ranging (LiDAR) and multispectral sensors. Structure-from-Motion (SfM) photogrammetry combined with interpolated Digital Terrain Models (DTMs) achieved strong agreement with LiDAR reference volumes (Root Mean Square Error (RMSE) ≈147 m 3 , Mean Absolute Percentage Error (MAPE) ≈2%). Sensitivity analysis confirmed that spatial resolution significantly influences accuracy, emphasizing the need for high-resolution data for precise volume estimation. This approach offers a scalable, cost-effective, and accurate alternative to conventional ground-based surveys. 3) Switchgrass Cultivation for Bioenergy and Water Quality Improvement We assessed the hydrological and water quality impacts of converting AMLs to switchgrass production areas using the Soil and Water Assessment Tool (SWAT). Results showed that converting 10% of the watershed area into the switchgrass production zone reduced streamflow by 3.1%, total suspended solids by 18.1%, total nitrogen by 7.6%, and total phosphorus by 6.2%, while achieving biomass yields of 8.6–9.2 metric tons per hectare. These findings highlight switchgrass as a dual-benefit strategy for land reclamation and bioenergy feedstock production. 4) Integrated Co-Firing and CCS for Carbon-Negative Power Generation We modeled co-firing scenarios using the Power Plant Flexible Model (PPFM) to evaluate plant efficiency, greenhouse gas (GHG) emissions, and levelized cost of electricity (LCOE). Without carbon capture and storage (CCS), increasing switchgrass co-firing ratios reduced LCOE from $\$$150/MWh at 0% biomass to $\$$110/MWh at full substitution. Under CCS, costs remained higher (~$\$$250/MWh at 0% biomass) but decreased to $\$$200/MWh at 100% biomass, while enabling net-zero or carbon-negative electricity due to switchgrass sequestration benefits. Although CCS introduced efficiency penalties, pairing it with biomass co-firing offset these impacts and maximized climate benefits. Overall, optimizing co-firing ratios between 60-100%, supported by reliable logistics and storage strategies, emerged as a practical pathway to balance affordability, sustainability, and net-zero or negative GHG emissions while promoting productive reuse of AMLs.

01 COAL, LIGNITE, AND PEAT↗

ATLAS (Automatic Tool for Local Assembly Structures) - A Comprehensive Infrastructure for Assembly, Annotation, and Genomic Binning of Metagenomic and Metaranscripomic Data

ATLAS (Automatic Tool for Local Assembly Structures) is a comprehensive multiomics data analysis pipeline that is massively parallel and scalable. ATLAS contains a modular analysis pipeline for assembly, annotation, quantification and genome binning of metagenomics and metatranscriptomics data and a framework for reference metaproteomic database construction. ATLAS transforms raw sequence data into functional and taxonomic data at the microbial population level and provides genome-centric resolution through genome binning. ATLAS provides robust taxonomy based on majority voting of protein coding open reading frames rolled-up at the contig level using modified lowest common ancestor (LCA) analysis. ATLAS provides robust taxonomy based on majority voting of protein coding open reading frames rolled-up at the contig level using modified lowest common ancestor (LCA) analysis. ATLAS is user-friendly, easy install through bioconda maintained as open-source on GitHub, and is implemented in Snakemake for modular customizable workflows.

genome binning↗

Predicting U 3 O 8 powder processing conditions: An AI/ML approach analyzing deep learning embeddings of SEM micrographs

High-resolution SEM images of uranium-oxide powders encode micro- and nanoscale clues to their synthesis route and calcination temperature. We trained a ResNet-50 model on 11 commercial-scale U₃O₈ classes, ammonium diuranate (ADU) or uranyl peroxide (H₂O₂) precursors calcined at temperatures ranging from 400 to 750 °C and added a 256-D projection head before the classifier to analyze the learned representation. The best of eight seeds reached 92.4 % accuracy on reserved testing data, but our focus is the structure of the embedding space rather than the accuracy and labels. We quantify class relatedness in the original 256-D space using centroid similarity and distributional distances, and we use Uniform Manifold Approximation Projection (UMAP) for visualization. ‘Unknown’ images from different preparation methods, SEM operators, and from the literature localized near the expected classes under a nearest-centroid analysis without retraining, as well as clustered in similar UMAP space. In conclusion, this embedding-centered workflow complements black-box classification by providing quantitative, similarity-based comparisons of U₃O₈ morphologies and reduces storage space by up to 98 % for image data used in millisecond vector search comparisons.

36 MATERIALS SCIENCE↗

KBKit: A Python Toolkit for Kirkwood–Buff Theory from Molecular Dynamics

Thermodynamic properties of liquid mixtures govern processes that range from drug delivery to energy storage, yet extracting these properties from molecular simulations remains challenging. Kirkwood–Buff (KB) theory offers a rigorous route by linking microscopic pair distribution functions to macroscopic free energies, but practical use of the theory has been hindered by two obstacles: (i) the long simulations needed to obtain well-converged Kirkwood-Buff integrals (KBIs) and (ii) the specialized corrections required to translate finite-size data to the thermodynamic limit. $\texttt{KBKit}$ is an open-source Python package that removes these barriers. It automatically computes KBIs and derived thermodynamic quantities from GROMACS input files, applies state-of-the-art finite-size corrections, and provides built-in diagnostic tools to quantify statistical uncertainty. Written with modern software-engineering practices—continuous integration, extensive unit testing, and thorough documentation—$\texttt{KBKit}$ is both reliable and easy to extend. By condensing complex KBI analysis into a few intuitive commands, $\texttt{KBKit}$ enables researchers to incorporate KB theory into routine simulation workflows and accelerate the discovery of solution-phase thermodynamics.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

High spatial resolution imaging of biological tissues using nanospray desorption electrospray ionization mass spectrometry

Mass spectrometry imaging (MSI) enables label-free spatial mapping of hundreds of biomolecules in tissue sections. This capability provides valuable information on tissue heterogeneity that is difficult to obtain using population-averaged assays. Despite substantial developments in both instrumentation and methodology, MSI of tissue samples at single-cell resolution remains challenging. In our research, we describe a protocol for robust imaging of tissue sections with a high (better than 10-μm) spatial resolution using nanospray desorption electrospray ionization (nano-DESI) mass spectrometry, an ambient ionization technique that does not require sample pretreatment before analysis. In this protocol, mouse uterine tissue is used as a model system to illustrate both the workflow and data obtained in these experiments. We provide a detailed description of the nano-DESI MSI platform, fabrication of the nano-DESI and shear force probes, shear force microscopy experiments, spectral acquisition, and data processing. A properly trained researcher (e.g., technician, graduate student, or postdoc) can complete all the steps from probe fabrication to data acquisition and processing within a single day. We further describe a new strategy for acquiring both positive- and negative-mode imaging data in the same experiment. This is achieved by alternating between positive and negative data acquisition modes during consecutive line scans. Using our imaging approach, hundreds of high-quality ion images were obtained from a single uterine section. This protocol enables sensitive and quantitative imaging of lipids and metabolites in heterogeneous tissue sections with high spatial resolution, which is critical to understanding biochemical processes occurring in biological tissues.

59 BASIC BIOLOGICAL SCIENCES↗

Hybrid Analysis of Fusion Data for Online Understanding of Complex Science on Extreme Scale Computers

The current practice for fusion scientists running first principle simulations on high performance computing plat-forms is to either run their simulations and output their data for post-hoc analysis, or to place in situ analytics into their code. In this paper we examine a complex workflow using XGC fusions simulation run on the Oak Ridge Leadership Computing Facility's supercomputer Summit, which also involve three anal-yses as part of the results necessary for scientific discovery. We discuss the challenges faced when implementing these algorithms and present an original hybrid staging technique to help enable the physicists to make discoveries during the execution of the simulation. By creating this infrastructure, we can examine complicated physics results, which may not have been possible without the infrastructure. For example, our work enables the online visualization of turbulent homoclinic tangle around the magnetic X-point, breaking the last confinement surface. This visualization could help fusion scientists to better understand and improve the turbulence spread of plasma exhaust heat, which is crucial toward realizing plasmas beyond the currently accessible physics regimes of present-day tokamak reactors. The physics of turbulent homoclinic tangle will be reported in a future physics publication, by utilizing the original online analysis/visualization framework presented in this paper.

Suchyta, Eric↗

Optimizing metaproteomics database construction: lessons from a study of the vaginal microbiome

Metaproteomics, a method for untargeted, high-throughput identification of proteins in complex samples, provides functional information about microbial communities and can tie functions to specific taxa. Metaproteomics often generates less data than other omics techniques, but analytical workflows can be improved to increase usable data in metaproteomic outputs. Identification of peptides in the metaproteomic analysis is performed by comparing mass spectra of sample peptides to a reference database of protein sequences. Although these protein databases are an integral part of the metaproteomic analysis, few studies have explored how database composition impacts peptide identification. Here, we used cervicovaginal lavage (CVL) samples from a study of bacterial vaginosis (BV) to compare the performance of databases built using six different strategies. We evaluated broad versus sample-matched databases, as well as databases populated with proteins translated from metagenomic sequencing of the same samples versus sequences from public repositories. Smaller sample-matched databases performed significantly better, driven by the statistical constraints on large databases. Additionally, large databases attributed up to 34% of significant bacterial hits to taxa absent from the sample, as determined orthogonally by 16S rRNA gene sequencing. We also tested a set of hybrid databases which included bacterial proteins from NCBI RefSeq and translated bacterial genes from the samples. These hybrid databases had the best overall performance, identifying 1,068 unique human and 1,418 unique bacterial proteins, ~30% more than a database populated with proteins from typical vaginal bacteria and fungi. Our findings can help guide the optimal identification of proteins while maintaining statistical power for reaching biological conclusions.

59 BASIC BIOLOGICAL SCIENCES↗

Consist v0.1.0

A Python library for provenance tracking, intelligent caching, and data virtualization in scientific simulation workflows. It automatically records code, configuration, and input data to skip redundant computations and enables querying results across many runs without manual bookkeeping. Designed to support multi-model simulation workflows like the BEAM CORE toolset at LBL, but designed to be extensible to a wide range of research workflows. Combines lineage tracking features as provided by OpenLineage with deterministic hashing like SnakeMake, and adds powerful analysis tools on model outputs.

Needell, Zachary [Lawrence Berkeley National Labor↗

Parallel File System I/O Performance Testing On LANL Clusters

These are slides from a presentation on parallel file system I/O performance testing on LANL clusters. I/O is a known bottleneck for HPC applications. Performance optimization of I/O is often required. This summer project entailed integrating IOR under Pavilion and automating the results analysis. The slides cover the following topics: scope of the work, tools utilized, IOR-Pavilion test workflow, build script, IOR parameters, how parameters are passed to IOR, *run_ior: functionality, Python IOR-Output Parser, Splunk data format, Splunk dashboard and features, and future work.

97 MATHEMATICS AND COMPUTING↗

Argonne Leadership Computing Facility (2018 Annual Report)

The Argonne Leadership Computing Facility enables breakthroughs in science and engineering by providing supercomputing resources and expertise to the research community. In 2018, we continued to build up programs and efforts that reinforce our overall simulation, learning, and data analysis capabilities as a facility. We also enhanced our services to better meet our users’ needs and workflow preferences.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Dynamic Measurements of the Structural Evolution of Material Defects at the Mesoscale

This is a technical report to be published as part of the FY2021 annual overview package. We have designed experiments and developed analytical methods using images from x-ray and electron microscopy in order to quantify the behavior of crystal dislocation defects at their native nanometer-length and micro-length scales. The results of these analyses are used to directly inform theoretical physics models and guide interpretation of dark field x-ray microscopy (DFXM) images that offer spatial resolution surpassing the resolution of beamline optics. Additionally, we developed new analytical methods to automate the alignment of optics hardware used in both synchrotron and x-ray free electron laser beamline facilities. The tools and workflows developed for this project are now being deployed as proof-of-concept in reaction history analysis (RHA) for future programmatic integration.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

A Dynamic Waste Isolation Pilot Plant Performance Assessment Tool - 12490

The Waste Isolation Pilot Plant (WIPP) Performance Assessment (PA) methodology comprises a toolbox used to demonstrate regulatory compliance of the repository after facility closure. The PA framework rests upon an extensive suite of computational codes. In some cases, significant alteration of code inputs is a tedious and difficult task. Due to the nature of the application for which they are used, PA codes used in support of WIPP regulatory compliance demonstration must satisfy stringent quality assurance requirements. Consequently, many of the coding practices used during original code development are still implemented today. A more efficient workflow configuration has the potential to alleviate difficulties associated with extensive code input modifications. Here, this potential is assessed via an implementation of a more flexible scientific workflow system for a subset of the codes used in WIPP PA. The scientific workflow approach taken here for a dynamic PA system enables users from disparate backgrounds to dramatically shorten the time between hypothesis and analysis by decreasing the amount of a priori knowledge, from a range of disciplines, needed to execute the code. Having smaller iteration times allows for more ideas to be tested and explored, which leads to safer and more optimized systems. Note that these high-level, dynamic tools are intended only for initial scoping studies on the personal computer of a researcher. Full, regulatory compliance calculations may occur only within a qualified computing environment. However, the WIPP PA tools here may guide future research and indicate regions of the analysis space that are worth further study. This next generation of PA software provides the ability to perform scoping investigations of repository performance quickly and easily, and has an accessible and useful interface to a variety of users, such as fuel cycle systems designers, domain experts such as repository modelers, and policy makers. The purview of this project allows for many opportunities for future work. Foremost among these is the desire to implement the full BRAGFLO suite within the workflow. This will entail porting or wrapping Genmesh, Matset, LHS, and ICSet within Python. Moreover, unifying the two GUIs into a single driver application would be a natural next step. Once the BRAGFLO suite is completed, other portions of WIPP PA could be implemented with corresponding and inter-operable work-flows. Likely first candidates for this are those codes that are similarly computationally intensive, such as the one used to generate complementary cumulative distribution functions used to demonstrate regulatory compliance (code CCDFGF). (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Model data for Flood Frequency Analysis using Stochastic Storm Transposition and an Integrated Surface-Subsurface Hydrological Model

This archived provides scripts and input files used for the implementation of a novel approach to conduct process-based Flood Frequency Analysis using a Stochastic Storm Transposition (SST) and an Integrated Surface-Subsurface Hydrological Model (ISSHM). As a proof-of-concept, this study uses the ISSHM, Advanced Terrestrial Simulator (Amanzi-ATS) model, and the SST model, RainyDay, to conduct flood frequency analysis by simulating the flood response to 5,000 annual synthetic storm events in a ~2000 km2 Southeast Texas watershed.The Watershed Workflow package is implemented in Python3. The Jupyter notebooks can be executed through multiple open-source tools, for example, Anaconda Jupyter Lab, VS Studio Code, etc. Other data files include TXT, CSV, DAT, SBATCH, SHP, TIF, NetCDF, and HDF5 files, which can be read through Python scripts. The input files for the ATS model and RainyDay model have .XML and .SST extensions, respectively, and can be edited in any commonly used text editors.This archive contains:* Scripts and data files essential for generating the ATS model input. It uses the Watershed Workflow package to produce both mesh and ATS input files. * Jupyter notebooks designated for the ATS model evaluation, covering both long-term simulations and 40 rainfall-runoff events.* Input files required to simulate SST storm events using RainyDay.

54 ENVIRONMENTAL SCIENCES↗

Simulating excited-state complex ensembles: Fluorescence and solvatochromism in amine-arene exciplexes

Exciplexes are excited-state complexes formed as a result of partial charge transfer from the donor to the acceptor species when one moiety of the donor–acceptor pair is electronically excited. The arene–amine exciplex formed between oligo-(p-phenylene) (OPP) and triethylamine (TEA) is of interest in the catalytic photoreduction of CO2 because it can compete with complete electron transfer to the OPP catalyst. Therefore, formation of the exciplex can hinder the generation of a radical anion OPP·− necessary for subsequent CO2 reduction. We report an implementation of a workflow automating quantum-chemistry calculations that generate and characterize an ensemble of structures to represent this exciplex state. We use FireWorks, Pymatgen, and Custodian Python packages for high-throughput ensemble generation. The workflow includes time-dependent density functional theory optimization, verification of excited-state minima, and exciplex characterization with natural transition orbitals, exciton analysis, excited-state Mulliken charges, and energy decomposition analysis. Fluorescence spectra computed for these ensembles using Boltzmann-weighted contributions of each structure agree better with experiment than our previous calculations based on a single representative exciplex structure [Kron et al., J. Phys. Chem. A 126, 2319–2329 (2022)]. The ensemble description of the exciplex state also reproduces an experimentally observed red shift of the emission spectrum of [OPP-4–TEA]* relative to [OPP-3–TEA]*. The workflow developed here streamlines otherwise labor-intensive calculations that would require significant user involvement and intervention.

Chemistry↗

Bioinformatics resource manager v2.3: an integrated software environment for systems biology with microRNA and cross-species analysis tools

Background: MicroRNAs (miRNAs) are noncoding RNAs that direct post-transcriptional regulation of protein coding genes. Recent studies have shown miRNAs are important for controlling many biological processes, including nervous system development, and are highly conserved across species. Given their importance, computational tools are necessary for analysis, interpretation and integration of high-throughput (HTP) miRNA data in an increasing number of model species. The Bioinformatics Resource Manager (BRM) v2.3 is a software environment for data management, mining, integration and functional annotation of HTP biological data. In this study, we report recent updates to BRM for miRNA data analysis and cross-species comparisons across datasets. Results: BRM v2.3 has the capability to query predicted miRNA targets from multiple databases, retrieve potential regulatory miRNAs for known genes, integrate experimentally derived miRNA and mRNA datasets, perform ortholog mapping across species, and retrieve annotation and cross-reference identifiers for an expanded number of species. Here we use BRM to show that developmental exposure of zebrafish to 30 uM nicotine from 6–48 hours post fertilization (hpf) results in behavioral hyperactivity in larval zebrafish and alteration of putative miRNA gene targets in whole embryos at developmental stages that encompass early neurogenesis. We show typical workflows for using BRM to integrate experimental zebrafish miRNA and mRNA microarray datasets with example retrievals for zebrafish, including pathway annotation and mapping to human ortholog. Functional analysis of differentially regulated (p<0.05) gene targets in BRM indicates that nicotine exposure disrupts genes involved in neurogenesis, possibly through misregulation of nicotine-sensitive miRNAs. Conclusions: BRM provides the ability to mine complex data for identification of candidate miRNAs or pathways that drive phenotypic outcome and, therefore, is a useful hypothesis generation tool for systems biology. The miRNA workflow in BRM allows for efficient processing of multiple miRNA and mRNA datasets in a single software environment with the added capability to interact with public data sources and visual analytic tools for HTP data analysis at a systems level. BRM is developed using Java™ and other open-source technologies for free distribution (http://www.sysbio.org/dataresources/brm.stm).

59 BASIC BIOLOGICAL SCIENCES↗

ProvLight: Efficient Workflow Provenance Capture on the Edge-to-Cloud Continuum

Modern scientific workflows require hybrid infrastructures combining numerous decentralized resources on the IoT/Edge interconnected to Cloud/HPC systems (aka the Computing Continuum) to enable their optimized execution. Understanding and optimizing the performance of such complex Edge-to-Cloud workflows is challenging. Capturing the provenance of key performance indicators, with their related data and processes, may assist in understanding and optimizing workflow executions. However, the capture overhead can be prohibitive, particularly in resource-constrained devices, such as the ones on the IoT/Edge.To address this challenge, based on a performance analysis of existing systems, we propose ProvLight, a tool to enable efficient provenance capture on the IoT/Edge. We leverage simplified data models, data compression and grouping, and lightweight transmission protocols to reduce overheads. We further integrate ProvLight into the E2Clab framework to enable workflow provenance capture across the Edge-to-Cloud Continuum. This integration makes E2Clab a promising platform for the performance optimization of applications through reproducible experiments.We validate ProvLight at a large scale with synthetic workloads on 64 real-life IoT/Edge devices in the FIT IoT LAB testbed. Evaluations show that ProvLight outperforms state-of-the-art systems like ProvLake and DfAnalyzer in resource-constrained devices. ProvLight is 26—37x faster to capture and transmit provenance data; uses 5—7x less CPU; 2x less memory; transmits 2x less data; and consumes 2—2.5x less energy. ProvLight [1] and E2Clab [2] are available as open-source tools.

Rosendo, Daniel↗