Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 505 records · Page 28

The LLNL nuclear data infrastructure for the GNDS data format

The next generation of nuclear data infrastructure tools at the Livermore National Laboratory (LLNL) consists of pipeline of codes that read and process nuclear data from evaluated files saved in the new GNDS (Generalised Nuclear Data Structure) nuclear data format. The processing code FUDGE (For Updating Data and Generating Evaluations) is at the front-end of this pipeline as it reads and process the evaluated data for use in downstream transport codes. FUDGE is Python based with C and C++ extensions for computationally intensive tasks. As is the case for the evaluated data, the processed output is also saved in the GNDS format and the GIDI+ API is provided as the interface between the processed data and the transport codes. GIDI+ is a C++ based suite of codes and it includes GIDI (General Interaction Data Interface), a library for reading and writing GNDS data, and MCGIDI which is the cross section lookup, and reaction and product distribution sampling interface between Monte Carlo transport codes and the GNDS data. GIDI provides methods for easy access to the multi-group processed GNDS data and this is demonstrated through its implementation in ARDRA, the LLNL deterministic transport code. The evaluation and sampling methods in MCGIDI are available as both CPU and GPU methods which facilitates the use of MCGIDI in both traditional CPU-based as well as the next generation mixed model computational architectures. This is demonstrated through the GIDI+ implementation in MERCURY, the LLNL Monte Carlo transport code. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

GeneLab: Overview of Challenges and Opportunities

The NASA GeneLab project capitalizes on multi-omic technologies to maximize the return on spaceflight experiments. To do this, GeneLab maintains a publicly accessible database (GLDS) that houses spaceflight and spaceflight relevant multi-omics data, and collaborates with NASA principal investigators and projects to generate additional omics data. GeneLab houses more than 200 transcriptomic, proteomic, metabolomic and epigenomic datasets from plant, animal and microbial experiments, with a growing number of these having been produced by the GeneLab sample processing lab. The GLDS contains rich metadata about each experiment and has recently integrated radiation dosimetery data from experiments flown on the Space Shuttle. GeneLab has also recently implemented an effort to present processed data in the GLDS in addition to the raw omics data. The processed data will enable interpretation of the data by a larger group of students, scientists and the general public. Standard pipelines for the transformation of raw data into visualizations were developed by four GeneLab Analysis Working Groups (animals, plants, microbes, multi-omics) comprised of over 100 scientists from NASA and academia. These pipelines are now being used by a group of bioinformatics interns to provide standard basic analysis of the data for incorporation into GLDS.

Galazka, Jonathan M.↗

GeneLab: Open Science for Life in Space

The NASA GeneLab project capitalizes on multi-omic technologies to maximize the return on spaceflight experiments. To do this, GeneLab maintains a publicly accessible database (GLDS) that houses spaceflight and spaceflight relevant multi-omics data, and collaborates with NASA principal investigators and projects to generate additional omics data. GeneLab houses more than 200 transcriptomic, proteomic, metabolomic and epigenomic datasets from plant, animal and microbial experiments, with a growing number of these having been produced by the GeneLab sample processing lab. The GLDS contains rich metadata about each experiment and has recently integrated radiation dosimetery data from experiments flown on the Space Shuttle. GeneLab has also recently implemented an effort to present processed data in the GLDS in addition to the raw omics data. The processed data will enable interpretation of the data by a larger group of students, scientists and the general public. Standard pipelines for the transformation of raw data into visualizations were developed by four GeneLab Analysis Working Groups (animals, plants, microbes, multi-omics) comprised of over 100 scientists from NASA and academia. These pipelines are now being used by a group of bioinformatics interns to provide standard basic analysis of the data for incorporation into GLDS.

Galazka, Jonathan M.↗

Automated Pneumothorax Diagnosis using Deep Neural Networks

Thoracic ultrasound can provide information leading to rapid diagnosis of pneumothorax with improved accuracy over the standard physical examination and with higher sensitivity than anteroposterior chest radiography. However, the clinical We have Furthermore, remote environments, such as the battlefield or deep-space exploration, may lack expertise for diagnosing developed an automated image interpretation pipeline for the analysis of thoracic ultrasound data and the classification of pneumothorax events to provide decision support in such situations. Our pipeline consists of image preprocessing, data augmentation, and deep learning architectures for medical diagnosis. In this work, we demonstrate that robust, accurate interpretation of chest images and video can be achieved using deep neural networks. A number of novel image processing techniques were employed to achieve this result. Affine transformations were applied for data augmentation. Hyperparameters were optimized for learning rate, dropout regularization, batch size, and epoch iteration by a sequential model-based Bayesian approach. In addition, we utilized pretrained architecturesinterpretation of a patient medical image is highly operator dependent. certain pathologies., applying transfer learning and fine-tuning techniques to fully connected layers. Our pipeline yielded binary classification validation accuracies of 98.3% for M-mode images and 99.8% with B-mode video frames.

US Army collaboration↗

A Tool for Automatic Data Distribution for CFD Applications on Structured Grids

Development of HPF versions of NPB and ARC3D has shown that HPF provides an efficient, concise way to express parallelism and to organize data traffic. The use of HPF, as noted in the papers, requires an intimate knowledge of the applications and a detailed analysis of data affinity, data movement, and data granularity. To simplify and accelerate the task of developing HPF versions of existing CFD applications we have designed and implemented ADAPT (Automatic Data Alignment and Placement Tool). ADAPT analyzes a CFD application working on a single structured grid and generates HPF TEMPLATE, (RE)DISTRIBUTION, ALIGNMENT, and INDEPENDENT directives. The directives can be generated on the nest level, subroutine level, application level, or on the application interface level. ADAPT annotates an existing CFD FORTRAN application, performing computations on single or multiple grids. On each grid the application is considered as a sequence of operators, each applied to a set of variables defined in a particular grid domain. ADAPT automatically detects implicit operators (i.e., having data dependences) and explicit operators (without data dependences). For parallelization of an explicit operator ADAPT creates a template for the operator domain, aligns arrays used in the operator with the template, distributes the template, and declares the loops over the distributed dimensions as INDEPENDENT. For parallelization of an implicit operator, the distribution of the operator's domain should be consistent with the operator's dependences. Any dependence between sections distributed on different processors would preclude parallelization if the compiler does not have an ability to pipeline computations. If a data distribution is "orthogonal" to the dependences of an implicit operator, then the loop which implements the operator can be declared as INDEPENDENT. ADAPT starts with an analysis of array index expressions of the loop nests. For each pair of arrays referenced in an assignment statement, it generates an arc in the alignment graph and annotates it with an affinity relation. The template, alignment, and distribution directives for a particular loop nest are then derived from a transitive closure of the affinity relation. A compromise of data distributions in different nests and subroutines is achieved by merging annotated alignment graphs for adjacent nests/stibroutine calls in the nest/call graph of the application in the process called distribution lifting. ADAPT has been implemented as a C++ program running in conjunction with a parallelization tool called CAPTools. ADAPT uses the parse tree, interprocedural analysis and application database generated by CAPTools. It also uses the Directed Graph class, initially implemented in p2d2 (parallel debugger oi distributed programs), and some other classes supporting symbolic computations. ADAPT uses data distribution techniques described. ADAPT was tested with ARC3D and the FT benchmark and has demonstrated a code performance within a factor of 1.5 of handwritten versions.

Frumkin, Michael↗

MetaboDirect: an analytical pipeline for the processing of FT-ICR MS-based metabolomic data

Background: Microbiomes are now recognized as the main drivers of ecosystem function ranging from the oceans and soils to humans and bioreactors. However, a grand challenge in microbiome science is to characterize and quantify the chemical currencies of organic matter (i.e., metabolites) that microbes respond to and alter. Critical to this has been the development of Fourier transform ion cyclotron resonance mass spectrometry (FT-ICR MS), which has drastically increased molecular characterization of complex organic matter samples, but challenges users with hundreds of millions of data points where readily available, user-friendly, and customizable software tools are lacking. Results: Here, we build on years of analytical experience with diverse sample types to develop MetaboDirect, an open-source, command-line-based pipeline for the analysis (e.g., chemodiversity analysis, multivariate statistics), visualization (e.g., Van Krevelen diagrams, elemental and molecular class composition plots), and presentation of direct injection high-resolution FT-ICR MS data sets after molecular formula assignment has been performed. When compared to other available FT-ICR MS software, MetaboDirect is superior in that it requires a single line of code to launch a fully automated framework for the generation and visualization of a wide range of plots, with minimal coding experience required. Among the tools evaluated, MetaboDirect is also uniquely able to automatically generate biochemical transformation networks (ab initio) based on mass differences (mass difference network-based approach) that provide an experimental assessment of metabolite connections within a given sample or a complex metabolic system, thereby providing important information about the nature of the samples and the set of microbial reactions or pathways that gave rise to them. Finally, for more experienced users, MetaboDirect allows users to customize plots, outputs, and analyses. Conclusion: Application of MetaboDirect to FT-ICR MS-based metabolomic data sets from a marine phage-bacterial infection experiment and a Sphagnum leachate microbiome incubation experiment showcase the exploration capabilities of the pipeline that will enable the research community to evaluate and interpret their data in greater depth and in less time. It will further advance our knowledge of how microbial communities influence and are influenced by the chemical makeup of the surrounding system. The source code and User’s guide of MetaboDirect are freely available through (https://github.com/Coayala/MetaboDirect) and (https://metabodirect.readthedocs.io/en/latest/), respectively.

54 ENVIRONMENTAL SCIENCES↗

The Caltech-NRAO Stripe 82 Survey (CNSS) Paper. I. The Pilot Radio Transient Survey in 50 Deg.(exp. 2)

We have commenced a multiyear program, the Caltech-NRAO Stripe 82 Survey (CNSS), to search for radio transients with the Jansky VLA in the Sloan Digital Sky Survey Stripe 82 region. The CNSS will deliver five epochs over the entire approx. 270 deg.(exp. 2) of Stripe 82, an eventual deep combined map with an rms noise of approx. 40 proper motion epoch y and catalogs at a frequency of 3 GHz, and having a spatial resolution of 3 inches. This first paper presents the results from an initial pilot survey of a 50 deg.(exp. 2) region of Stripe 82, involving four epochs spanning logarithmic timescales between 1 week and 1.5 yr, with the combined map having a median rms noise of 35 proper motion epoch y. This pilot survey enabled the development of the hardware and software for rapid data processing, as well as transient detection and follow-up, necessary for the full 270 deg.(exp. 2) survey. Data editing, calibration, imaging, source extraction, cataloging, and transient identification were completed in a semi-automated fashion within 6 hr of completion of each epoch of observations, using dedicated computational hardware at the NRAO in Socorro and custom-developed data reduction and transient detection pipelines. Classification of variable and transient sources relied heavily on the wealth of multiwavelength legacy survey data in the Stripe 82 region, supplemented by repeated mapping of the region by the Palomar Transient Factory. A total of 3.9(+0.5%/-0.9%) of the few thousand detected point sources werefound to vary by greater than 30%, consistent with similar studies at 1.4 and 5 GHz. Multiwavelength photometric data and light curves suggest that the variability is mostly due to shock-induced flaring in the jets of active galactic nuclei (AGNs). Although this was only a pilot survey, we detected two bona fide transients, associated with an RS CVn binary and a dKe star. Comparison with existing legacy survey data (FIRST, VLA-Stripe 82) revealed additional highly variable and transient sources on timescales between 5 and 20 yr, largely associated with renewed AGN activity. The rates of such AGNs possibly imply episodes of enhanced accretion and jet activity occurring once every approx. 40,000 yr in these galaxies. We compile the revised radio transient rates and make recommendations for future transient surveys and joint radio-optical experiments.

galaxies: active – radio continuum: galaxies –↗

Automatic Data Distribution for CFD Applications on Structured Grids

Development of HPF versions of NPB and ARC3D showed that HPF has potential to be a high level language for parallelization of CFD applications. The use of HPF requires an intimate knowledge of the applications and a detailed analysis of data affinity, data movement and data granularity. Since HPF hides data movement from the user even with this knowledge it is easy to overlook pieces of the code causing low performance of the application. In order to simplify and accelerate the task of developing HPF versions of existing CFD applications we have designed and partially implemented ADAPT (Automatic Data Distribution and Placement Tool). The ADAPT analyzes a CFD application working on a single structured grid and generates HPF TEMPLATE, (RE)DISTRIBUTION, ALIGNMENT and INDEPENDENT directives. The directives can be generated on the nest level, subroutine level, application level or inter application level. ADAPT is designed to annotate existing CFD FORTRAN application performing computations on single or multiple grids. On each grid the application can considered as a sequence of operators each applied to a set of variables defined in a particular grid domain. The operators can be classified as implicit, having data dependences, and explicit, without data dependences. In order to parallelize an explicit operator it is sufficient to create a template for the domain of the operator, align arrays used in the operator with the template, distribute the template, and declare the loops over the distributed dimensions as INDEPENDENT. In order to parallelize an implicit operator, the distribution of the operator's domain should be consistent with the operator's dependences. Any dependence between sections distributed on different processors would preclude parallelization if compiler does not have an ability to pipeline computations. If a data distribution is "orthogonal" to the dependences of an implicit operator then the loop which implements the operator can be declared as INDEPENDENT.

Frumkin, Michael↗

Development of Computational Environmental Microbiome Workflows for the Laboratory and the International Space Station

Identification of microorganisms in the spaceflight environment is critical for crew health risk assessment on the International Space Station (ISS). Since 2017, nanopore sequencing technology has been used to support thein situ identification of microbial species during spaceflight. Beginning in 2018, a culture-independent, swab-to-sequencer method was implemented onboard the ISS to provide a more thorough insight of the ISS microbiome. Eliminating microbial culture enables identification of difficult-to-culture organisms, reduces risks associated with potentially pathogenic cultures, and could significantly reduce the time from sample-to-answer. However, this molecular-based approach generates large metagenomic datasets that require substantial computational resources for analysis. To process nanopore-generated sequencing data, the JSC Microbiology Laboratory established a bioinformatics workflow on Amazon EC2 under the security guidance of the NASA Science Managed Cloud Environment (SMCE).This resource allows for the development, testing, and accessing of computational tools for processing large and complex datasets. The work described here will address the downlinking of data from the ISS, the automated pipeline developed to identify targeted bacterial and fungal organisms, and the time from sampling onboard to microbial identification. The pipelines have been enhanced to address high and low biomass samples using optimization based on sample source (air, water, or surface) and type of collection (filter, colony, or swab).The resulting microbiome data can be assessed beyond microbial identifications to gain understanding toward population changes over time, potential selective environmental pressures, and evaluating correlations with a wide range of additional data sets. Metagenome analysis pipelines in development could allow for simultaneous identification of microbial species, gene function, and gene pathways present in the environment. Beyond the ground processing, the developed analysis pipeline is currently deployed onboard the ISS to allow for near real-time assessments of the ISS microbiome. This study serves as a critical foundation for exploration missions, where rapid microbiome analyses will be required.

G. Marie Sharp↗

A laboratory breadboard system for dual-arm teleoperation

The computing architecture of a novel dual-arm teleoperation system is described. The novelty of this system is that: (1) the master arm is not a replica of the slave arm; it is unspecific to any manipulator and can be used for the control of various robot arms with software modifications; and (2) the force feedback to the general purpose master arm is derived from force-torque sensor data originating from the slave hand. The computing architecture of this breadboard system is a fully synchronized pipeline with unique methods for data handling, communication and mathematical transformations. The computing system is modular, thus inherently extendable. The local control loops at both sites operate at 100 Hz rate, and the end-to-end bilateral (force-reflecting) control loop operates at 200 Hz rate, each loop without interpolation. This provides high-fidelity control. This end-to-end system elevates teleoperation to a new level of capabilities via the use of sensors, microprocessors, novel electronics, and real-time graphics displays. A description is given of a graphic simulation system connected to the dual-arm teleoperation breadboard system. High-fidelity graphic simulation of a telerobot (called Phantom Robot) is used for preview and predictive displays for planning and for real-time control under several seconds communication time delay conditions. High fidelity graphic simulation is obtained by using appropriate calibration techniques.

Bejczy, A. K.↗

Fine-scale evaluation of two standard 16S rRNA gene amplicon primer pairs for analysis of total prokaryotes and archaeal nitrifiers in differently managed soils

The advance of high-throughput molecular biology tools allows in-depth profiling of microbial communities in soils, which possess a high diversity of prokaryotic microorganisms. Amplicon-based sequencing of 16S rRNA genes is the most common approach to studying the richness and composition of soil prokaryotes. To reliably detect different taxonomic lineages of microorganisms in a single soil sample, an adequate pipeline including DNA isolation, primer selection, PCR amplification, library preparation, DNA sequencing, and bioinformatic post-processing is required. Besides DNA sequencing quality and depth, the selection of PCR primers and PCR amplification reactions arguably have the largest influence on the results. This study tested the performance and potential bias of two primer pairs, i.e., 515F (Parada)-806R (Apprill) and 515F (Parada)-926R (Quince) in the standard pipelines of 16S rRNA gene Illumina amplicon sequencing protocol developed by the Earth Microbiome Project (EMP), against shotgun metagenome-based 16S rRNA gene reads. The evaluation was conducted using five differently managed soils. We observed a higher richness of soil total prokaryotes by using reverse primer 806R compared to 926R, contradicting to in silico evaluation results. Both primer pairs revealed various degrees of taxon-specific bias compared to metagenome-derived 16S rRNA gene reads. Nonetheless, we found consistent patterns of microbial community variation associated with different land uses, irrespective of primers used. Total microbial communities, as well as ammonia oxidizing archaea (AOA), the predominant ammonia oxidizers in these soils, shifted along with increased soil pH due to agricultural management. In the unmanaged low pH plot abundance of AOA was dominated by the acid-tolerant NS-Gamma clade, whereas limed agricultural plots were dominated by neutral-alkaliphilic NS-Delta/NS-Alpha clades. This study stresses how primer selection influences community composition and highlights the importance of primer selection for comparative and integrative studies, and that conclusions must be drawn with caution if data from different sequencing pipelines are to be compared.

16S rRNA gene amplicon Illumina sequencing↗

Enabling Early Transient Discovery in LSST via Difference Imaging with DECam

We present SLIDE, a pipeline that enables transient discovery in data from the Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST), using archival images from the Dark Energy Camera as templates for difference imaging. We apply this pipeline to the recently released Data Preview 1 (DP1; the first public release of Rubin commissioning data) and search for transients in the resulting difference images. The image subtraction, photometry extraction, and transient detection are all performed on the Rubin Science Platform. We demonstrate that SLIDE effectively extracts clean photometry by circumventing poor or missing LSST templates. We identified 29 previously unreported transients, 12 of which would not have been detected based on the DP1 DiaObject catalog. SLIDE will be especially useful for transient analysis in the early years of LSST, when template coverage will be largely incomplete or when templates may be contaminated by transients present at the time of acquisition. We present multiband light curves for a sample of known transients, along with new transient candidates identified through our search. Finally, we discuss the prospects of applying this pipeline during the main LSST survey. Our pipeline is broadly applicable and will support studies of all transients with slowly evolving phases.

Dong, Yize 一泽董 [Harvard-Smithsonian Center for Ast↗

A modeling pipeline to relate municipal wastewater surveillance and regional public health data

As COVID-19 becomes endemic, public health departments require improved passive indicators, which are independent of voluntary testing data, to estimate the prevalence of COVID-19 in local communities. Quantification of SARS-CoV-2 RNA from wastewater has the potential to be a powerful passive indicator. However, connecting measured SARS-CoV-2 RNA to community prevalence is challenging. We have developed a generalized methodology to improve the predictive power of wastewater measurements and applied it to data collected from treatment plants in the Chicago area. We built and compared a set of multi-linear regression models, which incorporate pepper mild mottle virus (PMMoV) as a population biomarker, Bovine coronavirus (BCoV) as a recovery control, and wastewater system flow rate into a corrected estimate for SARS-CoV-2 RNA concentration. For our data, models with BCoV performed better than those with PMMoV, but all model estimates of prevalence significantly improved correlation compared to doing no correction. We also investigate the utility of RNA measurements in wastewater as a leading indicator of COVID-19 trends. We do this in a rolling manner for corrected wastewater data and for other prevalence indicators, and statistically compare the temporal relationship between new increases in the wastewater data and those in other prevalence indicators. We find that wastewater trends often lead other COVID-19 indicators in predicting new surges.

60 APPLIED LIFE SCIENCES↗

The anomaly of the CMB power with the latest Planck data

Abstract The lack of power anomaly is an unexpected feature observed at large angular scales in the maps of Cosmic Microwave Background (CMB) produced by the COBE, WMAP andPlancksatellites. This signature, which consists in a missing of power with respect to that predicted by the ΛCDM model, might hint at a new cosmological phase before the standard inflationary era.The main point of this paper is taking into account the latestPlanckpolarisation data to investigate how the CMB polarisation improves the understanding of this feature. With this aim, we apply to the latestPlanckdata, both PR3 (2018) and PR4 (2020) releases, a new class of estimators capable of evaluating this anomaly by considering temperature and polarisation data both separately and in a jointly way. This is the first time that the PR4 dataset has been used to study this anomaly. To critically evaluate this feature, taking into account the residuals of known systematic effects present in thePlanckdatasets, we analyse the cleaned CMB maps using different combinations of sky masks, harmonic range and binning on the CMB multipoles.Our analysis shows that the estimator based only on temperature data confirms the presence of a lack of power with a lower-tail-probability (LTP), depending on the component separation method, ≤ 0.33% and ≤ 1.76% for PR3 and PR4, respectively. To our knowledge, the LTP≤ 0.33% for the PR3 dataset is the lowest one present in the literature obtained fromPlanck2018 data, considering thePlanckconfidence mask. We find significant differences between these two datasets when polarisation is taken into account most likely due to a different level of systematics. Especially, the analysis with PR3 data, unlike that with PR4, seems to point towards a lack of power at large scales also for polarisation.Moreover, we also show that for the PR3 dataset the inclusion of the subdominant polarisation information provides estimates that are less likely accepted in a ΛCDM cosmological model than the only-temperature analysis over the entire harmonic-range considered. In particular, at ℓ max = 26, we found that no simulation has a value as low as the data for all the pipelines.

Astronomy & Astrophysics↗

Cosmology with persistent homology: a Fisher forecast

Abstract Persistent homology naturally addresses the multi-scale topological characteristics of the large-scale structure as a distribution of clusters, loops, and voids. We apply this tool to the dark matter halo catalogs from theQuijotesimulations, and build a summary statistic for comparison with the joint power spectrum and bispectrum statistic regarding their information content on cosmological parameters and primordial non-Gaussianity. Through a Fisher analysis, we find that constraints from persistent homology are tighter for 8 out of the 10 parameters by margins of 13–50%. The complementarity of the two statistics breaks parameter degeneracies, allowing for a further gain in constraining power when combined. We run a series of consistency checks to consolidate our results, and conclude that our findings motivate incorporating persistent homology into inference pipelines for cosmological survey data.

Astronomy & Astrophysics↗

Cosmological constraints from density-split clustering in the BOSS CMASS galaxy sample

We present a clustering analysis of the BOSS DR12 CMASS galaxy sample, combining measurements of the galaxy two-point correlation function and density-split clustering down to a scale of $1 \, h^{-1}\, \text{Mpc}$. Our theoretical framework is based on emulators trained on high-fidelity mock galaxy catalogues that forward model the cosmological dependence of the clustering statistics within an extended-ΛCDM framework, including redshift-space and Alcock–Paczynski distortions. Our base-ΛCDM analysis finds ω cdm = 0.1201 ± 0.0022, σ 8 = 0.792 ± 0.034, and n s = 0.970 ± 0.018, corresponding to fσ 8 = 0.462 ± 0.020 at z ≈ 0.525, which is in agreement with Planck 2018 predictions and various clustering studies in the literature. We test single-parameter extensions to base-ΛCDM, varying the running of the spectral index, the dark energy equation of state, and the density of mass-less relic neutrinos, finding no compelling evidence for deviations from the base model. We model the galaxy–halo connection using a halo occupation distribution framework, finding signatures of environment-based assembly bias in the data. We validate our pipeline against mock catalogues that match the clustering and selection properties of CMASS, showing that we can recover unbiased cosmological constraints even with a volume 84 times larger than the one used in this study.

79 ASTRONOMY AND ASTROPHYSICS↗