Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
NASA NTRS2023

Prototype Global Analysis of LISA Data with Multiple Source Types

The novel data analysis challenges posed by the Laser Interferometer Space Antenna (LISA) arise from the overwhelmingly large number of astrophysical sources in the measurement band and the density with which they are found in the data. Robust detection and characterization of the numerous gravitational wave sources in LISA data can not be done sequentially, but rather through a simultaneous global fit of a data model containing the full suite of astrophysical and instrumental features present in the data. While previous analyses have focused on individual source types in isolation, here we present the first demonstration of a LISA global fit analysis containing combined astrophysical populations. The prototype pipeline uses a blocked Metropolis Hastings algorithm to alternatingly fit to a population of ultra compact galactic binaries, known “verification binaries” already identified by electromagnetic observations, a population of massive black hole mergers, and an instrument noise model. The Global LISA Analysis Software Suite (GLASS) is assembled from independently developed samplers for the different model components. The modular design enables flexibility to future development by defining standard interfaces for adding new, or updating additional, components to the global fit without being overly prescriptive for how those modules must be internally designed. The GLASS pipeline is demonstrated on data simulated for the LISA Data Challenge 2b. Results of the analysis and a road-map for continued development are described in detail.

Gravitational Waves↗
NASA NTRS1994

An application of miniscale experiments on Earth to refine microgravity analysis of adiabatic multiphase flow in space

Adiabatic two-phase flow is of interest to the design of multiphase fluid and thermal management systems for spacecraft. This paper presents original data and unifies existing data for capillary tubes as a step toward assessing existing multiphase flow analysis and engineering software. Comparisons of theory with these data once again confirm the broad accuracy of the theory. Due to the simplicity and low cost of the capillary tube experiments, which were performed on earth, we were able to closely examine for the first time a flow situation that had not previously been examined appreciably by aircraft tests. This is the situation of a slug flow at high quality, near transition to annular flow. Our comparison of software calculations with these data revealed overprediction of pipeline pressure drop by up to a factor of three. In turn, this finding motivated a reexamination of the existing theory, and then development of a new analytical and is in far better agreement with the data. This sequence of discovery illustrates the role of inexpensive miniscale modeling on earth to anticipate microgravity behavior in space and to complete and help define needs for aircraft tests.

Rothe, Paul H.↗
NASA NTRS2025

Mind the Gap: Addressing Data Gaps and Assessing Noise Mismodeling in LISA

Due to the sheer complexity of the Laser Interferometer Space Antenna (LISA) space mission, data gaps arising from instrumental irregularities and/or scheduled maintenance are unavoidable. Focusing on merger-dominated massive black hole binary signals, we test the appropriateness of the Whittle-likelihood on gapped data in a variety of cases. From first principles, we derive the likelihood valid for gapped data in both the time and frequency domains. Cheap-to-evaluate proxies to p-p plots are derived based on a Fisher-based formalism, and verified through Bayesian techniques. Our tools allow to predict the altered variance in the parameter estimates that arises from noise mismodeling, as well as the information loss represented by the broadening of the posteriors. The result of noise mismodeling with gaps is sensitive to the characteristics of the noise model, with strong low-frequency (red) noise and strong high-frequency (blue) noise giving statistically significant fluctuations in recovered parameters. We demonstrate that the introduction of a tapering window reduces statistical inconsistency errors, at the cost of less precise parameter estimates. We also show that the assumption of independence between inter-gap segments appears to be a fair approximation even if the data set is inherently coherent. However, if one instead assumes fictitious correlations in the data stream, when the data segments are actually independent, then the resultant parameter recoveries could be inconsistent with the true parameters. The theoretical and numerical practices that are presented in this work could readily be incorporated into global-fit pipelines operating on gapped data.

LISA↗
NASA NTRSDate not supplied

Tracking the Hunga Tonga-Hunga Ha’apai Eruption Stratospheric Aerosol and Trace Gas Plumes Using Machine Learning

On January 15, 2022, the Hunga Tonga-Hunga Ha’apai (hereafter, Hunga Tonga) submarine volcano had an explosive eruption that thrusted ash, gases, and water vapor through the troposphere into the stratosphere and mesosphere. Previous studies manually tracked the aerosol and trace gas plumes over time across different positions in the southern hemisphere. Using data retrieved from low earth orbiting satellite instruments (e.g., OMPS, OMI, and CALIPSO), this research demonstrates how open-source machine learning (ML) models, like Meta’s Segment Anything Model (SAM), with prompt engineering can perform automatic plume tracking following the Hunga Tonga eruption. This extensible methodology, and modular data processing and modeling pipeline using NASA Earthdata and Openscapes, establishes a framework for systematically and rapidly studying extreme events, including volcanic eruptions and large-scale wildfires. By combining advanced machine learning techniques, such as SAM’s zero-shot learning, with large volumes of remote sensing data, this work demonstrates how AI and open science can accelerate research and generate actionable results. The tools and technologies presented here can help translate earth science to action from NASA’s current and future Earth observing satellite missions (e.g., the Atmosphere Observing System (AOS)), and assist researchers and stakeholders in understanding, mapping, and responding to natural disasters and extreme events in a changing world.

David M. Giles↗
NASA NTRSDate not supplied

Evaluating Machine Learning Approaches to Plume Tracking

On July 15, 2022, the Hunga Tonga-Hunga Ha’apai (HTHH) submarine volcano erupted, propelling trace gasses and ash through the troposphere and up into the stratosphere. Previous studies manually tracked the aerosol and trace gas plumes over time across different positions in the southern hemisphere. Using imagery from NASA’s Earth Observing System, including MODIS aerosol products and OMI sulfur dioxide products, this research demonstrates how open-source machine learning (ML) models, like Meta’s Segment Anything Model (SAM), can perform automatic plume tracking following the Hunga Tonga eruption. This extensible methodology, and modular data processing and modeling pipeline, establishes a framework for systematically and rapidly studying natural disasters, including additional volcanic eruptions and large-scale wildfires. By combining advanced machine learning techniques, such as SAM’s zero-shot learning, with large volumes of NASA’s Earth Observation and remote sensing data, this work shows how AI and open science can accelerate research and generate actionable results, even for unprecedented events. The tools and technologies presented here can help translate earth science to action from NASA’s current and future Earth observing satellite missions, and assist researchers and stakeholders in understanding, mapping, and responding to natural disasters in a changing world.

machine learning↗
NASA NTRS2019

GeneLab: Overview of Challenges and Opportunities

The NASA GeneLab project capitalizes on multi-omic technologies to maximize the return on spaceflight experiments. To do this, GeneLab maintains a publicly accessible database (GLDS) that houses spaceflight and spaceflight relevant multi-omics data, and collaborates with NASA principal investigators and projects to generate additional omics data. GeneLab houses more than 200 transcriptomic, proteomic, metabolomic and epigenomic datasets from plant, animal and microbial experiments, with a growing number of these having been produced by the GeneLab sample processing lab. The GLDS contains rich metadata about each experiment and has recently integrated radiation dosimetery data from experiments flown on the Space Shuttle. GeneLab has also recently implemented an effort to present processed data in the GLDS in addition to the raw omics data. The processed data will enable interpretation of the data by a larger group of students, scientists and the general public. Standard pipelines for the transformation of raw data into visualizations were developed by four GeneLab Analysis Working Groups (animals, plants, microbes, multi-omics) comprised of over 100 scientists from NASA and academia. These pipelines are now being used by a group of bioinformatics interns to provide standard basic analysis of the data for incorporation into GLDS.

Galazka, Jonathan M.↗
NASA NTRS2019

GeneLab: Open Science for Life in Space

The NASA GeneLab project capitalizes on multi-omic technologies to maximize the return on spaceflight experiments. To do this, GeneLab maintains a publicly accessible database (GLDS) that houses spaceflight and spaceflight relevant multi-omics data, and collaborates with NASA principal investigators and projects to generate additional omics data. GeneLab houses more than 200 transcriptomic, proteomic, metabolomic and epigenomic datasets from plant, animal and microbial experiments, with a growing number of these having been produced by the GeneLab sample processing lab. The GLDS contains rich metadata about each experiment and has recently integrated radiation dosimetery data from experiments flown on the Space Shuttle. GeneLab has also recently implemented an effort to present processed data in the GLDS in addition to the raw omics data. The processed data will enable interpretation of the data by a larger group of students, scientists and the general public. Standard pipelines for the transformation of raw data into visualizations were developed by four GeneLab Analysis Working Groups (animals, plants, microbes, multi-omics) comprised of over 100 scientists from NASA and academia. These pipelines are now being used by a group of bioinformatics interns to provide standard basic analysis of the data for incorporation into GLDS.

Galazka, Jonathan M.↗
NASA NTRS2018

Automated Pneumothorax Diagnosis using Deep Neural Networks

Thoracic ultrasound can provide information leading to rapid diagnosis of pneumothorax with improved accuracy over the standard physical examination and with higher sensitivity than anteroposterior chest radiography. However, the clinical We have Furthermore, remote environments, such as the battlefield or deep-space exploration, may lack expertise for diagnosing developed an automated image interpretation pipeline for the analysis of thoracic ultrasound data and the classification of pneumothorax events to provide decision support in such situations. Our pipeline consists of image preprocessing, data augmentation, and deep learning architectures for medical diagnosis. In this work, we demonstrate that robust, accurate interpretation of chest images and video can be achieved using deep neural networks. A number of novel image processing techniques were employed to achieve this result. Affine transformations were applied for data augmentation. Hyperparameters were optimized for learning rate, dropout regularization, batch size, and epoch iteration by a sequential model-based Bayesian approach. In addition, we utilized pretrained architecturesinterpretation of a patient medical image is highly operator dependent. certain pathologies., applying transfer learning and fine-tuning techniques to fully connected layers. Our pipeline yielded binary classification validation accuracies of 98.3% for M-mode images and 99.8% with B-mode video frames.

US Army collaboration↗
NASA NTRS2000

A Tool for Automatic Data Distribution for CFD Applications on Structured Grids

Development of HPF versions of NPB and ARC3D has shown that HPF provides an efficient, concise way to express parallelism and to organize data traffic. The use of HPF, as noted in the papers, requires an intimate knowledge of the applications and a detailed analysis of data affinity, data movement, and data granularity. To simplify and accelerate the task of developing HPF versions of existing CFD applications we have designed and implemented ADAPT (Automatic Data Alignment and Placement Tool). ADAPT analyzes a CFD application working on a single structured grid and generates HPF TEMPLATE, (RE)DISTRIBUTION, ALIGNMENT, and INDEPENDENT directives. The directives can be generated on the nest level, subroutine level, application level, or on the application interface level. ADAPT annotates an existing CFD FORTRAN application, performing computations on single or multiple grids. On each grid the application is considered as a sequence of operators, each applied to a set of variables defined in a particular grid domain. ADAPT automatically detects implicit operators (i.e., having data dependences) and explicit operators (without data dependences). For parallelization of an explicit operator ADAPT creates a template for the operator domain, aligns arrays used in the operator with the template, distributes the template, and declares the loops over the distributed dimensions as INDEPENDENT. For parallelization of an implicit operator, the distribution of the operator's domain should be consistent with the operator's dependences. Any dependence between sections distributed on different processors would preclude parallelization if the compiler does not have an ability to pipeline computations. If a data distribution is "orthogonal" to the dependences of an implicit operator, then the loop which implements the operator can be declared as INDEPENDENT. ADAPT starts with an analysis of array index expressions of the loop nests. For each pair of arrays referenced in an assignment statement, it generates an arc in the alignment graph and annotates it with an affinity relation. The template, alignment, and distribution directives for a particular loop nest are then derived from a transitive closure of the affinity relation. A compromise of data distributions in different nests and subroutines is achieved by merging annotated alignment graphs for adjacent nests/stibroutine calls in the nest/call graph of the application in the process called distribution lifting. ADAPT has been implemented as a C++ program running in conjunction with a parallelization tool called CAPTools. ADAPT uses the parse tree, interprocedural analysis and application database generated by CAPTools. It also uses the Directed Graph class, initially implemented in p2d2 (parallel debugger oi distributed programs), and some other classes supporting symbolic computations. ADAPT uses data distribution techniques described. ADAPT was tested with ARC3D and the FT benchmark and has demonstrated a code performance within a factor of 1.5 of handwritten versions.

Frumkin, Michael↗
NASA NTRS2016

The Caltech-NRAO Stripe 82 Survey (CNSS) Paper. I. The Pilot Radio Transient Survey in 50 Deg.(exp. 2)

We have commenced a multiyear program, the Caltech-NRAO Stripe 82 Survey (CNSS), to search for radio transients with the Jansky VLA in the Sloan Digital Sky Survey Stripe 82 region. The CNSS will deliver five epochs over the entire approx. 270 deg.(exp. 2) of Stripe 82, an eventual deep combined map with an rms noise of approx. 40 proper motion epoch y and catalogs at a frequency of 3 GHz, and having a spatial resolution of 3 inches. This first paper presents the results from an initial pilot survey of a 50 deg.(exp. 2) region of Stripe 82, involving four epochs spanning logarithmic timescales between 1 week and 1.5 yr, with the combined map having a median rms noise of 35 proper motion epoch y. This pilot survey enabled the development of the hardware and software for rapid data processing, as well as transient detection and follow-up, necessary for the full 270 deg.(exp. 2) survey. Data editing, calibration, imaging, source extraction, cataloging, and transient identification were completed in a semi-automated fashion within 6 hr of completion of each epoch of observations, using dedicated computational hardware at the NRAO in Socorro and custom-developed data reduction and transient detection pipelines. Classification of variable and transient sources relied heavily on the wealth of multiwavelength legacy survey data in the Stripe 82 region, supplemented by repeated mapping of the region by the Palomar Transient Factory. A total of 3.9(+0.5%/-0.9%) of the few thousand detected point sources werefound to vary by greater than 30%, consistent with similar studies at 1.4 and 5 GHz. Multiwavelength photometric data and light curves suggest that the variability is mostly due to shock-induced flaring in the jets of active galactic nuclei (AGNs). Although this was only a pilot survey, we detected two bona fide transients, associated with an RS CVn binary and a dKe star. Comparison with existing legacy survey data (FIRST, VLA-Stripe 82) revealed additional highly variable and transient sources on timescales between 5 and 20 yr, largely associated with renewed AGN activity. The rates of such AGNs possibly imply episodes of enhanced accretion and jet activity occurring once every approx. 40,000 yr in these galaxies. We compile the revised radio transient rates and make recommendations for future transient surveys and joint radio-optical experiments.

galaxies: active – radio continuum: galaxies –↗
NASA NTRS1999

Automatic Data Distribution for CFD Applications on Structured Grids

Development of HPF versions of NPB and ARC3D showed that HPF has potential to be a high level language for parallelization of CFD applications. The use of HPF requires an intimate knowledge of the applications and a detailed analysis of data affinity, data movement and data granularity. Since HPF hides data movement from the user even with this knowledge it is easy to overlook pieces of the code causing low performance of the application. In order to simplify and accelerate the task of developing HPF versions of existing CFD applications we have designed and partially implemented ADAPT (Automatic Data Distribution and Placement Tool). The ADAPT analyzes a CFD application working on a single structured grid and generates HPF TEMPLATE, (RE)DISTRIBUTION, ALIGNMENT and INDEPENDENT directives. The directives can be generated on the nest level, subroutine level, application level or inter application level. ADAPT is designed to annotate existing CFD FORTRAN application performing computations on single or multiple grids. On each grid the application can considered as a sequence of operators each applied to a set of variables defined in a particular grid domain. The operators can be classified as implicit, having data dependences, and explicit, without data dependences. In order to parallelize an explicit operator it is sufficient to create a template for the domain of the operator, align arrays used in the operator with the template, distribute the template, and declare the loops over the distributed dimensions as INDEPENDENT. In order to parallelize an implicit operator, the distribution of the operator's domain should be consistent with the operator's dependences. Any dependence between sections distributed on different processors would preclude parallelization if compiler does not have an ability to pipeline computations. If a data distribution is "orthogonal" to the dependences of an implicit operator then the loop which implements the operator can be declared as INDEPENDENT.

Frumkin, Michael↗
NASA NTRSDate not supplied

Development of Computational Environmental Microbiome Workflows for the Laboratory and the International Space Station

Identification of microorganisms in the spaceflight environment is critical for crew health risk assessment on the International Space Station (ISS). Since 2017, nanopore sequencing technology has been used to support thein situ identification of microbial species during spaceflight. Beginning in 2018, a culture-independent, swab-to-sequencer method was implemented onboard the ISS to provide a more thorough insight of the ISS microbiome. Eliminating microbial culture enables identification of difficult-to-culture organisms, reduces risks associated with potentially pathogenic cultures, and could significantly reduce the time from sample-to-answer. However, this molecular-based approach generates large metagenomic datasets that require substantial computational resources for analysis. To process nanopore-generated sequencing data, the JSC Microbiology Laboratory established a bioinformatics workflow on Amazon EC2 under the security guidance of the NASA Science Managed Cloud Environment (SMCE).This resource allows for the development, testing, and accessing of computational tools for processing large and complex datasets. The work described here will address the downlinking of data from the ISS, the automated pipeline developed to identify targeted bacterial and fungal organisms, and the time from sampling onboard to microbial identification. The pipelines have been enhanced to address high and low biomass samples using optimization based on sample source (air, water, or surface) and type of collection (filter, colony, or swab).The resulting microbiome data can be assessed beyond microbial identifications to gain understanding toward population changes over time, potential selective environmental pressures, and evaluating correlations with a wide range of additional data sets. Metagenome analysis pipelines in development could allow for simultaneous identification of microbial species, gene function, and gene pathways present in the environment. Beyond the ground processing, the developed analysis pipeline is currently deployed onboard the ISS to allow for near real-time assessments of the ISS microbiome. This study serves as a critical foundation for exploration missions, where rapid microbiome analyses will be required.

G. Marie Sharp↗
NASA NTRS1990

A laboratory breadboard system for dual-arm teleoperation

The computing architecture of a novel dual-arm teleoperation system is described. The novelty of this system is that: (1) the master arm is not a replica of the slave arm; it is unspecific to any manipulator and can be used for the control of various robot arms with software modifications; and (2) the force feedback to the general purpose master arm is derived from force-torque sensor data originating from the slave hand. The computing architecture of this breadboard system is a fully synchronized pipeline with unique methods for data handling, communication and mathematical transformations. The computing system is modular, thus inherently extendable. The local control loops at both sites operate at 100 Hz rate, and the end-to-end bilateral (force-reflecting) control loop operates at 200 Hz rate, each loop without interpolation. This provides high-fidelity control. This end-to-end system elevates teleoperation to a new level of capabilities via the use of sensors, microprocessors, novel electronics, and real-time graphics displays. A description is given of a graphic simulation system connected to the dual-arm teleoperation breadboard system. High-fidelity graphic simulation of a telerobot (called Phantom Robot) is used for preview and predictive displays for planning and for real-time control under several seconds communication time delay conditions. High fidelity graphic simulation is obtained by using appropriate calibration techniques.

Bejczy, A. K.↗
NASA NTRS1981

Real-time multiple-look synthetic aperture radar processor for spacecraft applications

A spaceborne synthetic aperture radar (SAR) having pipeline multiple-look data processing is described which makes use of excessive azimuth bandwidth in radar echo signals to produce multiple-looking images. Time multiplexed single-look image lines from an azimuth correlator go through an energy analyzer which analyzes the mean energy in each separate look to determine the radar antenna electric boresight for use in generating the correct reference functions for the production of high quality SAR images. The multiplexed single look image lines also go through a registration delay to produce multi-look images.

Wu, C.↗
NASA NTRS1983

High-speed digital signal normalization for feature identification

A design approach for high speed normalization of digital signals was developed. A reciprocal look up table technique is employed, where a digital value is mapped to its reciprocal via a high speed memory. This reciprocal is then multiplied with an input signal to obtain the normalized result. Normalization improves considerably the accuracy of certain feature identification algorithms. By using the concept of pipelining the multispectral sensor data processing rate is limited only by the speed of the multiplier. The breadboard system was found to operate at an execution rate of five million normalizations per second. This design features high precision, a reduced hardware complexity, high flexibility, and expandability which are very important considerations for spaceborne applications. It also accomplishes a high speed normalization rate essential for real time data processing.

Ortiz, J. A.↗
NASA NTRS1984

Acoustooptic linear algebra processors - Architectures, algorithms, and applications

Architectures, algorithms, and applications for systolic processors are described with attention to the realization of parallel algorithms on various optical systolic array processors. Systolic processors for matrices with special structure and matrices of general structure, and the realization of matrix-vector, matrix-matrix, and triple-matrix products and such architectures are described. Parallel algorithms for direct and indirect solutions to systems of linear algebraic equations and their implementation on optical systolic processors are detailed with attention to the pipelining and flow of data and operations. Parallel algorithms and their optical realization for LU and QR matrix decomposition are specifically detailed. These represent the fundamental operations necessary in the implementation of least squares, eigenvalue, and SVD solutions. Specific applications (e.g., the solution of partial differential equations, adaptive noise cancellation, and optimal control) are described to typify the use of matrix processors in modern advanced signal processing.

Casasent, D.↗
NASA NTRS1986

A reliable pipelining protocol for the message service of the Mobile Satellite Experiment

This paper describes and analyzes a pipelining protocol for the data message communications of MSAT-X, a proposed experimental satellite-based mobile communications network. A demand assigned multiple access protocol using pure ALOHA for making reservation requests has been developed for MSAT-X under error-free assumptions. Preliminary propagation studies indicate that the short term bit error rate of satellite channels in a mobile environment can be as high as 0.001. Therefore, error-control schemes must be developed to ensure reliable transmissions. A retransmission scheme using selective repeat to minimize the end-to-end delay is proposed. Slotted ALOHA for making reservation requests is used to increase the overall system throughput. Since the number of channels available for reservation and data channels is essentially fixed for a given voice call blocking probability and a fixed call arrival rate, the analysis presented in this paper is also applicable to the integrated voice and data services of MSAT-X. Various operational scenarios have been investigated.

Li, V. O. K.↗