Search NASA⌕ Search

SEARCH · Search NASA

Results for “scripting”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 847 records · Page 47

Integration of TomoPy and the ASTRA toolbox for advanced processing and reconstruction of tomographic synchrotron data

The processing of tomographic synchrotron data requires advanced and efficient software to be able to produce accurate results in reasonable time. In this paper, the integration of two software toolboxes, TomoPy and the ASTRA toolbox, which, together, provide a powerful framework for processing tomographic data, is presented. The integration combines the advantages of both toolboxes, such as the user-friendliness and CPU-efficient methods of TomoPy and the flexibility and optimized GPU-based reconstruction methods of the ASTRA toolbox. It is shown that both toolboxes can be easily installed and used together, requiring only minor changes to existing TomoPy scripts. Furthermore, it is shown that the efficient GPU-based reconstruction methods of the ASTRA toolbox can significantly decrease the time needed to reconstruct large datasets, and that advanced reconstruction methods can improve reconstruction quality compared with TomoPy's standard reconstruction method.

Pelt, Daniël M.↗

Robust framework and software implementation for fast speciation mapping

One of the greatest benefits of synchrotron radiation is the ability to perform chemical speciation analysis through X-ray absorption spectroscopies (XAS). XAS imaging of large sample areas can be performed with either full-field or raster-scanning modalities. A common practice to reduce acquisition time while decreasing dose and/or increasing spatial resolution is to compare X-ray fluorescence images collected at a few diagnostic energies. In this work, several authors have used different multivariate data processing strategies to establish speciation maps. Furthermore, the theoretical aspects and assumptions that are often made in the analysis of these datasets are focused on. A robust framework is developed to perform speciation mapping in large bulk samples at high spatial resolution by comparison with known references. Two fully operational software implementations are provided: a user-friendly implementation within the MicroAnalysis Toolkit software, and a dedicated script developed under the R environment. The procedure is exemplified through the study of a cross section of a typical fossil specimen. Additionally, the algorithm provides accurate speciation and concentration mapping while decreasing the data collection time by typically two or three orders of magnitude compared with the collection of whole spectra at each pixel. Whereas acquisition of spectral datacubes on large areas leads to very high irradiation times and doses, which can considerably lengthen experiments and generate significant alteration of radiation-sensitive materials, this sparse excitation energy procedure brings the total irradiation dose greatly below radiation damage thresholds identified in previous studies. This approach is particularly adapted to the chemical study of heterogeneous radiation-sensitive samples encountered in environmental, material, and life sciences.

47 OTHER INSTRUMENTATION↗

Flexible Data-Aware Scheduling for Workflows over an In-Memory Object Store

This paper explores novel techniques for improving the performance of many-task workflows based on the Swift scripting language. We propose novel programmer options for automated distributed data placement and task scheduling. These options trigger a data placement mechanism used for distributing intermediate workflow data over the servers of Hercules, a distributed key-value store that can be used to cache file system data. We demonstrate that these new mechanisms can significantly improve the aggregated throughput of many-task workflows with up to 86x, reduce the contention on the shared file system, exploit the data locality, and trade off locality and load balance.

Duro, Francisco Rodrigo↗

Running Ensemble Workflows at Extreme Scale: Lessons Learned and Path Forward

The ever-increasing volumes of scientific data combined with sophisticated techniques for extracting information from them have led to the increasing popularity of ensemble workflows which are a collection of runs of individual workflows. A traditional approach followed by scientists to run ensembles is to rely on simple scripts to execute different runs and manage resources. This approach is not scalable and is error-prone, thereby motivating the development of workflow management systems that specialize in executing ensembles on HPC clusters. However, when the size of both the ensemble and the target system reach extreme scales, existing workflow management systems face new challenges that hamper their efficient execution. In this paper, we describe our experience scaling an ensemble workflow from the computational biology domain from the early design stages to the execution at extreme scale on Summit, a leadership class supercomputer at the Oak Ridge National Laboratory. We discuss challenges that arise when scaling ensembles to several million runs on thousands of HPC nodes. We identify challenges with composition of the ensemble itself, its execution at large scale, post-processing of the generated data, and scalability of the file system. Based on the experience acquired, we develop a generic vision of the capabilities and abstractions to add to existing workflow management systems to enable the execution of ensemble workflows at extreme scales. We believe that the understanding of these fundamental challenges will help application teams along with workflow system developers with designing the next generation of infrastructure for composing and executing extreme-scale ensemble workflows.

Mehta, Kshitij↗

Data Jockey: Automatic Data Management for HPC Multi-Tiered Storage Systems

We present the design and implementation of Data Jockey, a data management system for HPC multi-tiered storage systems. As a centralized data management control plane, Data Jockey automates bulk data movement and placement for scientific workflows and integrates into existing HPC storage infrastructures. Data Jockey simplifies data management by eliminating human effort in programming complex data movements, laying datasets across multiple storage tiers when supporting complex workflows, which in turn increases the usability of multitiered storage systems emerging in modern HPC data centers.Specifically, Data Jockey presents a new data management scheme called “goal driven data management” that can automatically infer low-level bulk data movement plans from declarative high-level goal statements that come from the lifetime of iterative runs of scientific workflows. While doing so, Data Jockey aims to minimize data wait times by taking responsibility for datasets that are unused or to be used, and aggressively utilizing the capacity of the upper, higher performant storage tiers.We evaluated a prototype implementation of Data Jockey under a synthetic workload based on a year’s worth of Oak Ridge Leadership Computing Facility’s (OLCF) operational logs. Our evaluations suggest that Data Jockey leads to higher utilization of the upper storage tiers while minimizing the programming effort of data movement compared to human involved, per-domain adhoc data management scripts.

Shin, Woong↗

HERCULES: A Pattern Driven Code Transformation System

New parallel computers are emerging, but developing efficient scientific code for them remains difficult. A scientist must manage not only the science-domain complexity but also the performance-optimization complexity. HERCULES is a code transformation system designed to help the scientist to separate the two concerns, which improves code maintenance, and facilitates performance optimization. The system combines three technologies, code patterns, transformation scripts and compiler plugins, to provide the scientist with an environment to quickly implement code transformations that suit his needs. Unlike existing code optimization tools, HERCULES is unique in its focus on user-level accessibility. In this paper we discuss the design, implementation and an initial evaluation of HERCULES.

Computer Science↗

SWIM: A Switch Manager for Datacenter Networks

As modern datacenter networks (DCNs) grow to support hundreds of thousands of servers and beyond, managing network equipment - such as routers, firewalls, and load balancers - becomes increasingly complex. Network attributes such as IP address allocations and BGP neighbor relations are scattered among various network engineering groups, which makes troubleshooting the network a cumbersome task. In addition, network vendor diversity leads to an explosion of vendor-specific management systems or single-use automation scripts, limiting network scalability while increasing the time required to perform management tasks. In this article, the authors propose a unified network management system, Switch Manager (SWIM), to cope with the growth by standardizing the language for describing network attributes and unifying the interface for executing management actions on the network equipment.

97 MATHEMATICS AND COMPUTING↗

Distribution Model Creation using Circuit Maps for Applied PV Impact Studies

Photovoltaic (PV) generation is becoming increasingly common on power distribution systems. The output fluctuations of this variable resource necessitate the creation of an accurate model to ensure that any adverse impacts to the distribution system are properly addressed. To reduce model creation time, a method of extracting data from AutoCAD drawings using scripted methods was developed. This paper will present a distribution model creation and validation process, document case studies of evaluating the impact of PV interconnections on utility distribution feeders, and discuss future applications. PV integration was found to be easier on 23 kV circuits compared to 4.16 kV circuits.

integrated circuit modeling↗

Modeling and Automation Framework for High IBR Integration in Large-Scale Power Systems

The increasing prevalence of power electronics- interfaced renewable generation sources is leading to a gradual replacement of traditional thermal generation-based synchronous machines. In this context, the modeling of a large-scale power grid that incorporates a significant number of inverter-based resources is crucial for understanding the dynamics and effects of these resources on the power system. This study investigates the positive sequence model of grid-following and grid-forming inverters. Additionally, this work explores the integration of distributed energy resources using population as an indicator of their relative geographic locations. To address challenge to integrate these inverter based resources into a realistic grid of the US Western interconnection, automation scripts are developed to streamline the process of replacing conventional generators with grid-following and grid-forming inverters, as well as allocating distributed energy resources. Different penetration levels of these inverters are considered, and their frequency regulation support following a disturbance is compared through dynamic simulations.

Lyu, Xue↗

Quantum/AI Topology-Aware Latency-Adaptive HPC Workflow Scheduling Optimization

The growing demand for more powerful high-performance computing (HPC) systems has led to a steady rise in energy consumption by supercomputing worldwide. This study is focused on comparing our Application-Topology Mapper (ATMapper) to the popular Simple Linux Utility for Resource Management (SLURM) for the purpose of exploring methods that can further optimize job-scheduling within HPC systems. ATMapper is an Artificial-Intelligence based approach to job-scheduling that is currently being enhanced with quantum annealing (QA) to generate optimal schedules faster. We are applying QA to speedup our ATMapper process to achieve higher computing efficiency, thereby reducing HPC energy consumption. Here, we examine how four job-scheduling approaches perform in processor node assignment when using an example network architecture of 4 interconnected nodes. Using a specialized script, we are assessing the schedule of a computation flow with 11 interdependent tasks. The data movements among nodes were tracked to count for the number of interactions (network hops) between nodes needed to complete the tasks. The total number of hops and the job completion time were then used to quantify the efficiency of the different mapping approaches. In addition to SLURM, we also compare our ATMapper to the QA-enabled LBNL TIGER and the D-Wave Distributed Computing processor assignment approaches. The preliminary results showed that our topology-aware, latency-adaptive ATMapper is significantly more efficient when compared to the other scheduling approaches due to its load-imbalance network allocation. The scheduler displayed a computing efficiency of 53% by performing significantly fewer network hops than its alternatives. By reducing the number of hops, ATMapper was able to perform all 11 tasks by using only 3 nodes out of given 4. This research indicates the potential to use QA/AI for HPC job-scheduling. Later, we will test a SLURM simulator program to draw further comparisons on the effectiveness of ATMapper's scheduling approach. The results of this comparison will serve as a baseline for later improving SLURM's performance using a QA-enhanced ATMapper approach.

Caraveo, Braulio [University of Huston - Clear Lak↗

Understanding the Performance and Potential of Cloud Computing for Scientific Applications

In this paper, commercial clouds bring a great opportunity to the scientific computing area. Scientific applications usually require significant resources, however not all scientists have access to sufficient high-end computing systems, may of which can be found in the Top500 list. Cloud Computing has gained the attention of scientists as a competitive resource to run HPC applications at a potentially lower cost. But as a different infrastructure, it is unclear whether clouds are capable of running scientific applications with a reasonable performance per money spent. This work studies the performance of public clouds and places this performance in context to price. We evaluate the raw performance of different services of AWS cloud in terms of the basic resources, such as compute, memory, network and I/O. We also evaluate the performance of the scientific applications running in the cloud. This paper aims to assess the ability of the cloud to perform well, as well as to evaluate the cost of the cloud running scientific applications. We developed a full set of metrics and conducted a comprehensive performance evlauation over the Amazon cloud. We evaluated EC2, S3, EBS and DynamoDB among the many Amazon AWS services. We evaluated the memory sub-system performance with CacheBench, the network performance with iperf, processor and network performance with the HPL benchmark application, and shared storage with NFS and PVFS in addition to S3. We also evaluated a real scientific computing application through the Swift parallel scripting system at scale. Armed with both detailed benchmarks to gauge expected performance and a detailed monetary cost analysis, we expect this paper will be a recipe cookbook for scientists to help them decide where to deploy and run their scientific applications between public clouds, private clouds, or hybrid clouds.

97 MATHEMATICS AND COMPUTING↗

STEPS: A Portable Numerical Simulation Toolkit for Electrical Power System Dynamic Studies

Numerical simulation is the key technique for large scale power system analysis. Redistribution of global renewable power via international interconnections requires new simulation tools to study the interconnected systems with different nominal frequencies as a whole. In this paper we introduce an open source simulation toolkit for electrical power systems (STEPS) which is hosted at Github. Its kernel is coded in C++ with major functions of power flow and electro-mechanical dynamic simulation. Flexible options are provided and configurable to improve power flow solution and dynamic simulation. Common devices and models are supported in STEPS for AC/DC hybrid system studies. Studies of interconnected systems with different nominal frequencies is supported in STEPS for research of international interconnection. Application program interfaces are provided and wrapped with Python to enable high-level interfaces for general applications. STEPS is thread safe and parallel computation is supported in both kernel and script levels to accelerate simulation. It is portable and works on Windows and GNU/Linux platforms. Cases from small to large scale systems are thoroughly tested to validate the toolkit with commercial packages as benchmarks.

42 ENGINEERING↗

Genotyping‐in‐Thousands by sequencing (GT‐seq): A cost effective SNP genotyping method based on custom amplicon sequencing

Abstract Genotyping‐in‐Thousands by sequencing ( GT ‐seq) is a method that uses next‐generation sequencing of multiplexed PCR products to generate genotypes from relatively small panels (50–500) of targeted single‐nucleotide polymorphisms ( SNP s) for thousands of individuals in a single Illumina HiSeq lane. This method uses only unlabelled oligos and PCR master mix in two thermal cycling steps for amplification of targeted SNP loci. During this process, sequencing adapters and dual barcode sequence tags are incorporated into the amplicons enabling thousands of individuals to be pooled into a single sequencing library. Post sequencing, reads from individual samples are split into individual files using their unique combination of barcode sequences. Genotyping is performed with a simple perl script which counts amplicon‐specific sequences for each allele, and allele ratios are used to determine the genotypes. We demonstrate this technique by genotyping 2068 individual steelhead trout ( Oncorhynchus mykiss ) samples with a set of 192 SNP markers in a single library sequenced in a single Illumina HiSeq lane. Genotype data were 99.9% concordant to previously collected TaqMan ™ genotypes at the same 192 loci, but call rates were slightly lower with GT ‐seq (96.4%) relative to Taqman (99.0%). Of the 192 SNP s, 187 were genotyped in ≥90% of the individual samples and only 3 SNP s were genotyped in <70% of samples. This study demonstrates amplicon sequencing with GT ‐seq greatly reduces the cost of genotyping hundreds of targeted SNP s relative to existing methods by utilizing a simple library preparation method and massive efficiency of scale.

Campbell, Nathan R.↗

Soils' dirty little secret: Depth–based comparisons can be inadequate for quantifying changes in soil organic carbon and other mineral soil properties

Quantifying changes in soil organic carbon (SOC) stocks and other soil properties is essential for understanding how soils will respond to land management practices and global change. Although they are widely used, comparisons of SOC stocks at fixed depth (FD) intervals are subject to errors when changes in bulk density or soil organic matter occur. The equivalent soil mass (ESM) method has been recommended in lieu of FD for assessing changes in SOC stocks in mineral soils, but ESM remains underutilized for SOC stocks and has rarely been used for other soil properties. In this paper, we draw attention to the limitations of the FD method and demonstrate the advantages of the ESM approach. We provide illustrations to show that the FD approach is susceptible to errors not only for quantifying SOC stocks but also for soil mass-based properties such as SOC mass percent, C:N mass ratio, and δ 13 C. Here we describe the ESM approach and show how it mitigates the FD method limitations. Using bulk density change simulations applied to an empirical dataset from bioenergy cropping systems, we show that the ESM method provides consistently lower errors than FD when quantifying changes in SOC stocks and other soil properties. To simplify the use of ESM, we detail how the method can be integrated into sampling schemes, and we provide an example R computer script that can perform ESM calculations on large datasets. We encourage future studies, whether temporal or comparative, to utilize sampling methods that are amenable to the ESM approach. Overall, we agree with previous recommendations that ESM should be the standard method for evaluating SOC stock changes in mineral soils, but we further suggest that ESM may also be preferred for comparisons of other soil properties including mass percentages, elemental mass ratios, and stable isotope composition.

59 BASIC BIOLOGICAL SCIENCES↗

Tundra Trait Team: A database of plant traits spanning the tundra biome

Motivation: The Tundra Trait Team (TTT) database includes field-based measurements of key traits related to plant form and function at multiple sites across the tundra biome. This dataset can be used to address theoretical questions about plant strategy and trade-offs, trait-environment relationships and environmental filtering, and trait variation across spatial scales, to validate satellite data, and to inform earth system model parameters. Main types of variable contained: The database contains 81,537 measurements of 18 plant traits. The most frequently measured traits (>1,000 observations each) include plant height, leaf area, specific leaf area, leaf fresh and dry mass, leaf dry matter content, leaf nitrogen content, leaf carbon content, leaf phosphorus content, seed mass, and stem specific density. Spatial location and grain: Measurements were collected in tundra habitats in both the northern and southern hemispheres, including Arctic sites in Alaska, Canada, Greenland, Fennoscandia, and Siberia and alpine sites in the European Alps, Colorado Rockies, Caucasus, Ural Mountains, Pyrenees, Australian Alps, and Central Otago Mountains (New Zealand). More than 99% of observations are georeferenced. Time period and grain: All data were collected between 1964 and 2016. A small number of sites have repeated trait measurements at two or more time periods. Major taxa and level of measurement: Trait measurements were made on 858 terrestrial vascular plant species growing in tundra habitats. Most observations are on individuals (87%), while a small number represent plot or site means or maximums per species. Software format: csv file and GitHub repository with data cleaning scripts in R; contribution to TRY plant trait database (www.try-db.org) to be included in the next version release.

59 BASIC BIOLOGICAL SCIENCES↗

Global root traits (GRooT) database

Motivation: Trait data are fundamental to the quantitative description of plant form and function. Although root traits capture key dimensions related to plant responses to changing environmental conditions and effects on ecosystem processes, they have rarely been included in large-scale comparative studies and global models. For instance, root traits remain absent from nearly all studies that define the global spectrum of plant form and function. Thus, to overcome conceptual and methodological roadblocks preventing a widespread integration of root trait data into large-scale analyses we created the Global Root Trait (GRooT) Database. GRooT provides readyto- use data by combining the expertise of root ecologists with data mobilization and curation. Specifically, we (a) determined a set of core root traits relevant to the description of plant form and function based on an assessment by experts, (b) maximized species coverage through data standardization within and among traits, and (c) implemented data quality checks. Main types of variables contained: GRooT contains 114,222 trait records on 38 continuous root traits. Spatial location and grain: Global coverage with data from arid, continental, polar, temperate and tropical biomes. Data on root traits were derived from experimental studies and field studies. Time period and grain: Data were recorded between 1911 and 2019. Major taxa and level of measurement: GRooT includes root trait data for which taxonomic information is available. Trait records vary in their taxonomic resolution, with subspecies or varieties being the highest and genera the lowest taxonomic resolution available. It contains information for 184 subspecies or varieties, 6,214 species, 1,967 genera and 254 families. Owing to variation in data sources, trait records in the database include both individual observations and mean values. Software format: GRooT includes two csv files. A GitHub repository contains the csv files and a script in R to query the database.

59 BASIC BIOLOGICAL SCIENCES↗

High-throughput sequencing of the chloroplast and mitochondrion of Chlamydomonas reinhardtii to generate improved de novo assemblies, analyze expression patterns and transcript speciation, and evaluate diversity among laboratory strains and wild isolates

Chlamydomonas reinhardtii is a unicellular chlorophyte alga that is widely studied as a reference organism for understanding photosynthesis, sensory and motile cilia, and for development of an algal-based platform for producing biofuels and bio-products. Its highly repetitive, ~205-kbp circular chloroplast genome and ~15.8-kbp linear mitochondrial genome were sequenced prior to the advent of high-throughput sequencing technologies. Here, high coverage shotgun sequencing was used to assemble both organellar genomes de novo. These new genomes correct dozens of errors in the prior genome sequences and annotations. Gen-ome sequencing coverage indicates that each cell contains on average 83 copies of the chloroplast genome and 130 copies of the mitochondrial genome. Using protocols and analyses optimized for organellar tran-scripts, RNA-Seq was used to quantify their relative abundances across 12 different growth conditions. Forty-six percent of total cellular mRNA is attributable to high expression from a few dozen chloroplast genes. RNA-Seq data were used to guide gene annotation, to demonstrate polycistronic gene expression, and to quantify splicing of psaA and psbA introns. In contrast to a conclusion from a recent study, we found that chloroplast transcripts are not edited. Unexpectedly, cytosine-rich polynucleotide tails were observed at the 3’-end of all mitochondrial transcripts. A comparative genomics analysis of eight laboratory strains and 11 wild isolates of C. reinhardtii identified 2658 variants in the organellargenomes, which is 1/10th as much genetic diversity as is found in the nucleus.

ftsH↗