Search NASA⌕ Search

SEARCH · Search NASA

Results for “analysis workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

Pufferscale: Rescaling HPC Data Services for High Energy Physics Applications

User-space HPC data services are emerging as an appealing alternative to traditional parallel file systems, because of their ability to be tailored to application needs while eliminating unnecessary overheads incurred by POSIX compliance. The High Energy Physics (HEP) community is progressively turning towards such services to enable high-throughput accesses under heavy concurrency to billions of event data produced by instruments and consumed by subsequent analysis workflows. Such services would benefit from the possibility to be rescaled up and down to adapt to changing workloads, as experimental campaigns progress, in order to optimize resource usage. This paper formalizes rescaling a distributed storage system as a multi objective optimization problem considering three criteria: load balance, data balance, and duration of the rescaling operation. We propose a heuristic for rapidly finding a good approximate solution, while allowing users to weight the criteria as needed. The heuristic is evaluated with Pufferscale, a new rescaling manager for microservice-based distributed storage systems. To validate our approach in a real-world ecosystem, we showcase the use of Pufferscale as a means to enable storage malleability in the HEPnOS storage system for HEP applications.

Distributed Storage System↗

Combining In-situ and In-transit Processing to Enable Extreme-Scale Scientific Analysis

With the onset of extreme-scale computing, I/O constraints make it increasingly difficult for scientists to save a sufficient amount of raw simulation data to persistent storage. One potential solution is to change the data analysis pipeline from a post-process centric to a concurrent approach based on either in-situ or in-transit processing. In this context computations are considered in-situ if they utilize the primary compute resources, while in-transit processing refers to offloading computations toa set of secondary resources using asynchronous data transfers. In this paper we explore the design and implementation of three common analysis techniques typically performed on large-scale scientific simulations: topological analysis, descriptive statistics, and visualization. We summarize algorithmic developments, describe a resource scheduling system to coordinate the execution ofvarious analysis workflows, and discuss our implementation using the DataSpaces and ADIOS frameworks that support efficient data movement between in-situ and in-transit computations. Wedemonstrate the efficiency of our lightweight, flexible framework by deploying it on the Jaguar XK6 to analyze data generated by S3D, a massively parallel turbulent combustion code. Our framework allows scientists dealing with the data deluge at extreme scale to perform analyses at increased temporal resolutions, mitigate I/O costs, and significantly improve the time to insight.

Bennett, Janine C.↗

Performance of the reconstruction of large impact parameter tracks in the inner detector of ATLAS

Searches for long-lived particles (LLPs) are among the most promising avenues for discovering physics beyond the Standard Model at the Large Hadron Collider (LHC). However, displaced signatures are notoriously difficult to identify due to their ability to evade standard object reconstruction strategies. In particular, the ATLAS track reconstruction applies strict pointing requirements which limit sensitivity to charged particles originating far from the primary interaction point. To recover efficiency for LLPs decaying within the tracking detector volume, the ATLAS Collaboration employs a dedicated large-radius tracking (LRT) pass with loosened pointing requirements. During Run 2 of the LHC, the LRT implementation produced many incorrectly reconstructed tracks and was therefore only deployed in small subsets of events. In preparation for LHC Run 3, ATLAS has significantly improved both standard and large-radius track reconstruction performance, allowing for LRT to run in all events. This development greatly expands the potential phase-space of LLP searches and streamlines LLP analysis workflows. This paper will highlight the above achievement and report on the readiness of the ATLAS detector for track-based LLP searches in Run 3.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

NV: Nessus Vulnerability Visualization for the Web

Network vulnerability is a critical component of network se- curity. Yet vulnerability analysis has received relatively lit- tle attention from the security visualization community. In this paper we describe nv, a web-based Nessus vulnerability visualization. Nv utilizes treemaps and linked histograms to allow system administrators to discover, analyze, and man- age vulnerabilities on their networks. In addition to visual- izing single Nessus scans, nv supports the analysis of sequen- tial scans by showing which vulnerabilities have been fixed, remain open, or are newly discovered. Nv was also designed to operate completely in-browser, to avoid sending sensitive data to outside servers. We discuss the design of nv, as well as provide case studies demonstrating vulnerability analysis workflows which include a multiple-node testbed and data from the 2011 VAST Challenge.

information visualization↗

STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality—even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7 × faster than SZ3.

Wang, Daoce [University of Nebraska, Omaha]↗

Whole Energy Homes

This repository contains analysis workflows, associated python packages, and documentation of the Whole Energy Homes study. A reproducible Snakemake workflow for data processing, feature extraction, and modeling using the weh python package. Built with Copier from the able-workflow-copier template.

Pathak, Maharshi [Northeastern Univ., Boston, MA (↗

Machine Learning for Distributed Acoustic Sensing data (MLDAS) v1.0.1

MLDAS is a Python-written package for exploratory data analysis and deep learning training on Distributed Acoustic Sensing data. The machine learning tools are powered by the PyTorch library and designed to work efficiently on large scale datasets using parallel computing. Various SLURM scripts as well as a tutorial have also been made available to allow geophysicists to quickly and easily implement the available tools in their analysis workflow on supercomputer facilities.

Dumont, Vincent↗

Massively Parallel and Portable Genomic Sequence Analysis

Massively Parallel and Portable Genomic Sequence Analysis (mappgene) is a sequencing analysis workflow for high performance computing. It incorporates novel technologies to simplify and accelerate genetics research.

Moon, Josephy↗

NOODLES [SWR-22-78]

NOODLES is a protocol specification for collaborative visualization. It allows software tools of any type to participate in an analysis or visualization session. Use cases include, but are not limited to, distributed analysis, workflow interoperability, computational steering, etc.

Brunhart-Lupo, Nicholas↗

acdc – Automated Contamination Detection and Confidence estimation for single-cell genome data

Abstract Background A major obstacle in single-cell sequencing is sample contamination with foreign DNA. To guarantee clean genome assemblies and to prevent the introduction of contamination into public databases, considerable quality control efforts are put into post-sequencing analysis. Contamination screening generally relies on reference-based methods such as database alignment or marker gene search, which limits the set of detectable contaminants to organisms with closely related reference species. As genomic coverage in the tree of life is highly fragmented, there is an urgent need for a reference-free methodology for contaminant identification in sequence data. Results We present acdc, a tool specifically developed to aid the quality control process of genomic sequence data. By combining supervised and unsupervised methods, it reliably detects both known and de novo contaminants. First, 16S rRNA gene prediction and the inclusion of ultrafast exact alignment techniques allow sequence classification using existing knowledge from databases. Second, reference-free inspection is enabled by the use of state-of-the-art machine learning techniques that include fast, non-linear dimensionality reduction of oligonucleotide signatures and subsequent clustering algorithms that automatically estimate the number of clusters. The latter also enables the removal of any contaminant, yielding a clean sample. Furthermore, given the data complexity and the ill-posedness of clustering, acdc employs bootstrapping techniques to provide statistically profound confidence values. Tested on a large number of samples from diverse sequencing projects, our software is able to quickly and accurately identify contamination. Results are displayed in an interactive user interface. Acdc can be run from the web as well as a dedicated command line application, which allows easy integration into large sequencing project analysis workflows. Conclusions Acdc can reliably detect contamination in single-cell genome data. In addition to database-driven detection, it complements existing tools by its unsupervised techniques, which allow for the detection of de novo contaminants. Our contribution has the potential to drastically reduce the amount of resources put into these processes, particularly in the context of limited availability of reference species. As single-cell genome data continues to grow rapidly, acdc adds to the toolkit of crucial quality assurance tools.

59 BASIC BIOLOGICAL SCIENCES↗

ESS-DIVE Reporting Format for Amplicon Abundance Table

While standardized sequencing data is available in public repositories and efforts such as MIxS for common sample collection and processing metadata are well established, the lack of common bioinformatic processing metadata has hindered the ability to do large-scale metaanalyses and the potential for data re-use by non-experts such as ecosystem, watershed, or earth system modelers. To address this need for Department of Energy researchers, we have developed an amplicon reporting format which captures both sample preparation and bioinformatic processing metadata and stores processed amplicon data as a paired abundance table and sequencing file to maximize the potential for re-use of these data. To aid in the adoption of accessible and reproducible analysis workflows, this reporting format was developed in concert with amplicon functionality within the Department of Energy’s Systems Biology Knowledgebase (KBase) to ensure common data and metadata requirements and facilitate seamless transfer between these platforms.This dataset contains support documentation for the amplicon reporting format (README.md and instructions.md), templates for both bioinformatic and sequencing metadata (amplicon_bioinformatic_metadata_template_2021_10_03.csv and amplicon_sequencing_metadata_template_2021_10_03.csv), a crosswalk indicating how this reporting format relates to the current MIxS format (ESSDIVE-MIxS_crosswalk.csv), a list of available instrument terms (amplicon_seq_instrument_terms_2021_10_03.csv), a map between QIIME2 parameter settings and metadata fields (amplicon_qiime2_plugin_metadata_map.csv), a data dictionary (amplicon_CSV_dd.csv), and file-level metadata (amplicon_FLMD.csv).

54 ENVIRONMENTAL SCIENCES↗

Depth-resolved sagebrush root metabolomics, rhizosphere microbial communities, and geochemistry at the East River Watershed

This data set consists of results from soil nutrient profile, untargeted metabolomics, mass spec imaging, and amplicon sequencing. Data for soil nutrient profile includes common cations (Ca, Mg, Na, and K etc.) extracted from 3 digesting steps – ammonia acetate (for exchangeable cations), nitric acid (for acid dissolved fraction), and hydrofluoric acid/perchloric acid (HF/HClO4) for whole soil digestion. It also includes concentration of organic carbon, inorganic nitrogen (ammonia and nitrate) and phosphorus (Bray-1 P and nitric acid extract), and total nitrogen and phosphorus. Data for untargeted metabolomics includes metabolomic profile for root exudate/tissues and soil extracts from depths at surface soil to saprolite, that were measured using gas chromatography – mass spectrometry (GC-MS), and liquid chromatography – tandem mass spectrometry (LC-MS/MS). Data for mass spec imaging includes spatial distribution of metabolites that were detected and annotated with Fourier transformation ion cyclotron resonance mass spectrometer (FTICR-MS). Data for amplicon sequencing includes the base paired 16S and ITS ribosomal RNA sequences from Miseq Illumina sequencing. All samples were collected from 2 sampling campaign October 2022 and June 2023. Collectively, these datasets enable a mechanistic evaluation of how nutrient acquisition, especially nitrogen and phosphorus, differs between shallow roots operating in soil and deep roots functioning within the fractured bedrock zone. All files are provided as comma-separated values (CSV) fies (.csv) and (GZIP) file (.gz). The compressed .gz FASTQ files can be read directly in R using the dada2 package as part of the amplicon sequence analysis workflow. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231. This research was performed on a project award 60563 (https://dx.doi.org/10.46936/expl.proj.2022.60563/60008727) from the Environmental Molecular Sciences Laboratory, a DOE Office of Science User Facility sponsored by the Biological and Environmental Research program under Contract No. DE-AC05-76RL01830.

EARTH SCIENCE > AGRICULTURE > SOILS > CARBON↗

Small Angle Scattering Data Analysis Assisted by Machine Learning Methods

Small angle scattering (SAS) is a widely used technique for characterizing structures of wide ranges of materials. For such wide ranges of applications of SAS, there exist a large number of ways to model the scattering data. While such analysis models are often available from various suites of SAS data analysis software packages, selecting the right model to start with poses a big challenge for beginners to SAS data analysis. Here, we present machine learning (ML) methods that can assist users by suggesting scattering models for data analysis. A series of one-dimensional scattering curves have been generated by using different models to train the algorithms. The performance of the ML method is studied for various types of ML algorithms, resolution of the dataset, and the number of the dataset. The degree of similarities among selected scattering models is presented in terms of the confusion matrix. The scattering model suggestions with prediction scores provide a list of scattering models that are likely to succeed. Therefore, if implemented with extensive libraries of scattering models, this method can speed up the data analysis workflow by reducing search spaces for appropriate scattering models.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework

Scientific applications are increasingly using cloud resources for their data analysis workflows. However, managing data effectively and efficiently over these cloud resources is challenging due to the myriad storage choices with different performance, cost trade-offs, complex application choices and complexity associated with elasticity, failure rates in these environments. The different data access patterns for data-intensive scientific applications require a more flexible and robust data management solution than the ones currently in existence. FRIEDA is a Flexible Robust Intelligent Elastic Data Management framework that employs a range of data management strategies in cloud environments. FRIEDA can manage storage and data lifecycle of applications in cloud environments. There are four different stages in the data management lifecycle of FRIEDA – (i) storage planning, (ii) provisioning and preparation, (iii) data placement, and (iv) execution. FRIEDA defines a data control plane and an execution plane. The data control plane defines the data partition and distribution strategy, whereas the execution plane manages the execution of the application using a master-worker paradigm. FRIEDA also provides different data management strategies, either to partition the data in real-time, or predetermine the data partitions prior to application execution.

97 MATHEMATICS AND COMPUTING↗

Department of Energy's Biological and Environmental Research Strategic Data Roadmap for Earth System Science

Rapid advances in experimental, sensor, and computational technologies and techniques are driving exponential growth in the volume, acquisition rate, variety, and complexity of scientific data. This wealth of scientifically meaningful data has tremendous potential to lead to scientific discovery. However, to achieve scientific breakthroughs, these data must be exploitable—they must be analyzed effectively and efficiently and the results shared and communicated easily within the wider Department of Energy’s (DOE’s) Biological and Environmental Research (BER) Climate and Environmental Sciences Division (CESD) community. The explosion in data complexity and scale makes these tasks exceedingly difficult to achieve, particularly given that an increasing number of disciplines are working across techniques, integrating simulation and experimental or observational results (see Table 5 in Appendix 2). Consequently, we need new approaches to data management, analysis, and visualization that provide research teams with easy-to-use and scalable end-to-end solutions. These solutions must facilitate (and where feasible, automate and capture) every stage in the data lifecycle (shown in Figure 1), from collection to management, annotation, sharing, discovery, analysis, and visualization. In addition, the core functionalities are the same across climate science communities, but they require customization to adapt to specific needs and fit into research and analysis workflows. To this end, the mission of CESD’s Data and Informatics Program is to integrate all existing and future distributed CESD data holdings into a seamless and unified environment for the acceleration of Earth system science.

54 ENVIRONMENTAL SCIENCES↗

DOE High Performance Computing Operational Review (HPCOR): Enabling Data-Driven Scientific Discovery at HPC Facilities

U.S. Department of Energy (DOE) High Performance Computing (HPC) facilities are on the verge of a paradigm shift in the way they deliver systems and services to science and engineering teams. Research projects are producing a wide variety of data at unprecedented scale and level of complexity, with community-specific services that are part of the data collection and analysis workflow. On June 18-19, 2014 representatives from six DOE HPC centers met in Oakland, CA at the DOE High Performance Operational Review (HPCOR) to discuss how they can best provide facilities and services to enable large-scale data-driven scientific discovery at the DOE national laboratories. The report contains findings from that review.

97 MATHEMATICS AND COMPUTING↗

Information Theoretic Measures for Visual Analytics: The Silver Ticket?

In the context of text-based analysis workflows, we propose that an effective analytic tool facilitates triage by a) enabling users to identify and set aside irrelevant content (i.e., reduce the complexity of information in a dataset) and b) develop a working mental model of which items are most relevant to the question at hand. This LDRD funded research developed a dataset that is enabling this team to evaluate propose normalized compression distance (NCD) as a task, user, and context-insensitive measure of categorization outcomes (Shannon entropy is reduced as order is imposed). Effective analytics tools help people impose order, reducing complexity in measurable ways. Our concept and research was documented in a paper accepted to the ACM conference "Beyond Time and Error: Novel Methods in Information Visualization Evaluation", part of the IEEE VisWeek Conference, Baltimore, MD, October 16-21, 2016. The paper is included as an appendix to this report.

97 MATHEMATICS AND COMPUTING↗

Evaluating Symmetry in 3-dimensional Simulation Data

The increasing use of three-dimensional simulations has led to an increased need for effective verification and validation, or V&V. Unfortunately, the high computational cost associated with three-dimensional simulations makes many standard V&V techniques cost-prohibitive. Given these circumstances, we must pay close attention to V&V metrics that can be evaluated using only a single simulation. One of the simplest of these metrics is the evaluation of physical symmetries. This report details a method of computing the deviation from symmetry in a three-dimensional dataset with an arbitrarily distributed mesh. Examples are presented for spherical and axisymmetric systems using the Paraview post-processing analysis workflow.

97 MATHEMATICS AND COMPUTING↗