Search NASA⌕ Search

SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15
All sourcesNASA NTRSDOE OSTI
NASA NTRS2019

Citizen Science Twitter Data Management for Earth Science Applications

Social media data can provide useful real-time and historical information relating to the natural world, but managing this data poses challenges. Scientists at GES DISC are exploring the potential of Twitter data to augment precipitation data from the Global Precipitation Measurement (GPM) mission. However, the format of Twitter data is unconventional in the context of NASA data centers, resulting in frustration for scientists who need to work with the data. This study investigated procedures and standards needed to properly manage Twitter data to make them compatible with these data centers. After comparing databases, the study found that the MongoDB database was best suited for the storage of raw Twitter data due to its flexibility, ability to be accessed by multiple users, and querying functionality. The study used the Python package Zarr to transform processed Twitter data into a gridded format similar to that of satellite data. Each Tweet was mapped onto a time-space grid; each grid location contained information about Tweet attributes and precipitation. The study developed a pipeline for downloading, storing, and gridding Twitter data and transformed Twitter data into an understandable format for users of NASA satellite data.

Li, Rachel↗
NASA NTRS1993

The initial data products from the EUVE software - A photon's journey through the End-to-End System

The End-to-End System (EES) is a unique collection of software modules created for use at the Center for EUV Astrophysics. The 'pipeline' is a shell script which executes selected EES modules and creates initial data products: skymaps, data sets for individual sources (called 'pigeonholes') and catalogs of sources. This article emphasizes the data from the all-sky survey, conducted between July 22, 1992 and January 21, 1993. A description of each of the major data products will be given and, as an example of how the pipeline works, the reader will follow a photon's path through the software pipeline into a pigeonhole. These data products are the primary goal of the EUVE all-sky survey mission, and so their relative importance for the follow-up science will also be discussed.

Antia, Behram↗
NASA NTRS2016

Erasing the Milky Way: New Cleaning Technique Applied to GBT Intensity Mapping Data

We present the first application of a new foreground removal pipeline to the current leading HI intensity mapping dataset, obtained by the Green Bank Telescope (GBT). We study the 15- and 1-h field data of the GBT observations previously presented in Masui et al. (2013) and Switzer et al. (2013), covering about 41 square degrees at 0.6 less than z is less than 1.0, for which cross-correlations may be measured with the galaxy distribution of the WiggleZ Dark Energy Survey. In the presented pipeline, we subtract the Galactic foreground continuum and the point source contamination using an independent component analysis technique (fastica), and develop a Fourier-based optimal estimator to compute the temperature power spectrum of the intensity maps and cross-correlation with the galaxy survey data. We show that fastica is a reliable tool to subtract diffuse and point-source emission through the non-Gaussian nature of their probability distributions. The temperature power spectra of the intensity maps is dominated by instrumental noise on small scales which fastica, as a conservative sub-traction technique of non-Gaussian signals, can not mitigate. However, we determine similar GBT-WiggleZ cross-correlation measurements to those obtained by the Singular Value Decomposition (SVD) method, and confirm that foreground subtraction with fastica is robust against 21cm signal loss, as seen by the converged amplitude of these cross-correlation measurements. We conclude that SVD and fastica are complementary methods to investigate the foregrounds and noise systematics present in intensity mapping datasets.

Wolz, L.↗
NASA NTRS2019

The Cumulus Ecosystem: Open Source and Beyond to Foster Collaboration

NASA’s Earth Observing System Data and Information System (EOSDIS) open source Cumulus software is designed as a common set of code and services that can be used to create a pipeline to deliver and manage earth science data in the cloud. Cumulus strives to create an ecosystem on the foundation of open source that unites those with shared problems and goals by encouraging users to contribute solutions back to the platform. Large parts of ingesting and managing data are common and much of what is created can be used by others. Our goal is to maximize collaboration and code reuse while allowing users to design a custom solution that meets their needs without having to take on extraneous functionality. In this talk we will describe how the Cumulus ecosystem works beyond just open source software. We will review the technology, the successes and challenges, and the evolution and future of Cumulus as an ecosystem.

Cumulus↗
NASA NTRS2013

The Hopkins Ultraviolet Telescope: The Final Archive

The Hopkins Ultraviolet Telescope (HUT) was a 0.9 m telescope and moderate-resolution (Delta)lambda equals 3 A) far-ultraviolet (820-1850 Å) spectrograph that flew twice on the space shuttle, in 1990 December (Astro-1, STS-35) and 1995 March (Astro-2, STS-67). The resulting spectra were originally archived in a nonstandard format that lacked important descriptive metadata. To increase their utility, we have modified the original datareduction software to produce a new and more user-friendly data product, a time-tagged photon list similar in format to the Intermediate Data Files (IDFs) produced by the Far Ultraviolet Spectroscopic Explorer calibration pipeline. We have transferred all relevant pointing and instrument-status information from locally-archived science and engineering databases into new FITS header keywords for each data set. Using this new pipeline, we have reprocessed the entire HUT archive from both missions, producing a new set of calibrated spectral products in a modern FITS format that is fully compliant with Virtual Observatory requirements. For each exposure, we have generated quicklook plots of the fully-calibrated spectrum and associated pointing history information. Finally, we have retrieved from our archives HUT TV guider images, which provide information on aperture positioning relative to guide stars, and converted them into FITS-format image files. All of these new data products are available in the new HUT section of the Mikulski Archive for Space Telescopes (MAST), along with historical and reference documents from both missions. In this article, we document the improved data-processing steps applied to the data and show examples of the new data products.

Hopkins↗
NASA NTRS2018

Transcriptomics and Proteomics Discussion

This presentation will cover the the basic pipelines for transcriptomics and proteomics that the GeneLab Analysis Working Groups (AWGs) have so far determined to be optimal. Basic transcriptomic pipelines will first be presented from primary analysis to higher-order systems analysis. Examples of how the data has been analyzed will be presented. Proteomics pipelines will also be presented compiled from various AWG members. Discussion will be generated from the AWG members to reach a consensus for each omic type.

Transcriptomics↗
NASA NTRS1997

On the Floating Point Performance of the i860 Microprocessor

The i860 microprocessor is a pipelined processor that can deliver two double precision floating point results every clock. It is being used in the Touchstone project to develop a teraflop computer by the year 2000. With such high computational capabilities it was expected that memory bandwidth would limit performance on many kernels. Measured performance of three kernels showed performance is less than what memory bandwidth limitations would predict. This paper develops a model that explains the discrepancy in terms of memory latencies and points to some problems involved in moving data from memory to the arithmetic pipelines.

Lee, King↗
NASA NTRSDate not supplied

Transcriptomics Processing Pipelines for Space Biology: An Open Source and Consensus-Driven Approach

Transcriptomics holds significant value in elucidating the relationship between gene expression, experimental factors, biological factors, and various types of omics data. Enhancing our understanding of these connections is paramount for foundational biology, which plays a pivotal role in devising solutions for challenges pertinent to both space travel and terrestrial life. The NASA GeneLab project, part of the Open Science Data Repository (OSDR.nasa.gov), seeks to accelerate space biology research through cataloging and democratizing ‘omics data, including transcriptomics. Since raw omics data are largely inaccessible to non-bioinformaticians, GeneLab works with the scientific community via the Open Science Analysis Working Groups (AWGs) to develop standard processing pipelines to generate and publish processed data. Unlike raw data, processed data have greater immediate value to diverse users with varying technical backgrounds and computational capabilities. Standardizing processing workflows is essential to match the pace of raw data generation, ensure reproducibility, and enable standardized processed data for comparison across datasets. As of June 2023, transcriptomics studies comprise over half of GeneLab datasets hosted on the OSDR, including data from bulk RNA-seq and Affymetrix or Agilent 1-Channel DNA microarray assays. In collaboration with the AWGs, GeneLab developed consensus processing pipelines for these transcriptomics data types that includes quality control, background correction (microarray only), data normalization and quantification, culminating in the detection and annotation of differentially expressed genes. The work presented here describes Nextflow implementations of GeneLab’s consensus transcriptomics pipelines that automates and accelerates processing of these datasets. In addition to the core data processing, these workflows also include raw data staging and a robust verification and validation program to identify errors in real-time, stop additional downstream computation, and preserve computational resources. These workflows are used to generate GeneLab processed data hosted on the OSDR, and are publicly available as open source software for others to use at: https://github.com/nasa/GeneLab_Data_Processing.

Jonathan Oribello↗
NASA NTRS1984

Pipelined digital SAR azimuth correlator using hybrid FFT-transversal filter

A synthetic aperture radar system (SAR) having a range correlator is provided with a hybrid azimuth correlator which utilizes a block-pipe-lined fast Fourier transform (FFT). The correlator has a predetermined FFT transform size with delay elements for delaying SAR range correlated data so as to embed in the Fourier transform operation a corner-turning function as the range correlated SAR data is converted from the time domain to a frequency domain. The azimuth correlator is comprised of a transversal filter to receive the SAR data in the frequency domain, a generator for range migration compensation and azimuth reference functions, and an azimuth reference multiplier for correlation of the SAR data. Following the transversal filter is a block-pipelined inverse FFT used to restore azimuth correlated data in the frequency domain to the time domain for imaging.

Wu, C.↗
NASA NTRS2019

GEONEX Data Products: Geostationary Satellite Derived Land Surface and Atmospheric Public Data Products

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GEONEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GEONEX pipeline will begin to produce provisional data products to be consumed by external collaborators and the academic community. In order to better inform and introduce the GEONEX products to the science community, the provisional products shall be distributed from the NAS data portal, located at data.nas.nasa.gov, and simple webpage at www.nasa.gov/geonex has been deployed, which describes any algorithms used in deriving the products, user manuals and data file information. We will also update the status of the data processing, on the website and provide links to the latest datasets, and use a geonex mailing list, using lists.nasa.gov.

Wang, Weile↗
NASA NTRS1980

Outline of a fast hardware implementation of Winograd's DFT algorithm

The main characteristics of the discrete Fourier transform (DFT) algorithm considered by Winograd (1976) is a significant reduction in the number of multiplications. Its primary disadvantage is a higher structural complexity. It is, therefore, difficult to translate the reduced number of multiplications into faster execution of the DFT by means of a software implementation of the algorithm. For this reason, a hardware implementation is considered in the current study, taking into account a design based on the algorithm prescription discussed by Zohar (1979). The hardware implementation of a FORTRAN subroutine is proposed, giving attention to a pipelining scheme in which 5 consecutive data batches are being operated on simultaneously, each batch undergoing one of 5 processing phases.

Zohar, S.↗
NASA NTRS2004

Searching for Pulsations from CEN X-4

The goal of this proposal is to detect or place sensitive upper limits on pulsations from the quiescent neutron-star X-ray binary Cen X-4. The observations were obtained on 1 March 2003. Unfortunately, due to problems with the standard pipeline processing of this observation, the data were not delivered until more than one year later, on 21 April 2004. No pulsations have yet been found in the data set. However, new optical spectra have been obtained which will lead to better constraints on the orbital period of Cen X-4 and which may enable detection of pulsations.

Kaaret, Philip↗
NASA NTRS2005

The Nature of the Enigmatic 10-Minute Accreting Binary System ES CET

ES Cet is one of the most compact binary systems known with an orbital period of only 10.3 minutes. Our allocated observations with the XMM-Newton X-ray satellite were performed in January and July 2004, with the data being delivered to the PI in August 2004. Preliminary results were presented by the PI in September 2004 and January 2005. We have also secured supporting optical observations of ES Ceti using the Magellan telescopes (November 2004). The team is currently performing a thorough and final analysis of the X-ray, UV and optical data sets with the latest XMM pipeline software and our own analysis packages.

Steeghs, Daniel↗
NASA NTRS2014

The Spitzer Survey of Interstellar Clouds in the Gould Belt. VI. The Auriga-California Molecular Cloud Observed with IRAC and MIPS

We present observations of the Auriga-California Molecular Cloud (AMC) at 3.6, 4.5, 5.8, 8.0, 24, 70 and 160 micrometers observed with the IRAC and MIPS detectors as part of the Spitzer Gould Belt Legacy Survey. The total mapped areas are 2.5 deg(exp 2) with IRAC and 10.47 deg2 with MIPS. This giant molecular cloud is one of two in the nearby Gould Belt of star-forming regions, the other being the Orion A Molecular Cloud (OMC). We compare source counts, colors and magnitudes in our observed region to a subset of the SWIRE data that was processed through our pipeline. Using color-magnitude and color-color diagrams, we find evidence for a substantial population of 166 young stellar objects (YSOs) in the cloud, many of which were previously unknown. Most of this population is concentrated around the LkH(alpha) 101 cluster and the filament extending from it. We present a quantitative description of the degree of clustering and discuss the fraction of YSOs in the region with disks relative to an estimate of the diskless YSO population. Although the AMC is similar in mass, size and distance to the OMC, it is forming about 15 - 20 times fewer stars.

Orion A Molecular Cloud (OMC)↗
NASA NTRS2019

Creating Data-Driven Vector Visualizations of Satellite Orbit Tracks Using NASA GIBS and Worldview

NASA Earth Observing System (EOS) currently operates dozens of remote sensing satellites, many of which can be viewed directly in NASA’s open-source Worldview application. Much of this satellite imagery can be viewed in near-real time as it is processed and served by NASA’s Global Imagery Browse Service (GIBS). To better educate users on the time and location of imagery, GIBS serves orbit track specific layers for each satellite. Worldview has historically served these layers as raster images but recent updates have enabled the application to now serve these layers using vector tiles. With the release of Worldview v3.0, orbit track layers can be displayed using mapbox vector tiles (MVT). This visualization format allows users to not only view and change the color of orbit track layers, as they could do previously with rasters, but also inspect individual vector points and filter layers by specific parameters such as time. The data contained within a MVT is further enhanced in Worldview with the combination of a JSON description file served from GIBS used to describe the MVT data. This presentation will provide an overview of the process of consuming orbit track vector tiles and data files from GIBS using a pipeline to configure, build and ultimately display the orbit tracks in Worldview. Furthermore, the presentation aims to describe how others can leverage our open-source code to display and enhance vector layers in their own applications.

Rice, Zachary↗
NASA NTRS2019

A Chromaticity Analysis and PSF Subtraction Techniques for SCExAO/CHARIS Data

We present an analysis of instrument performance using new observations taken with the Coronagraphic High Angular Resolution Imaging Spectrograph (CHARIS) instrument and the Subaru Coronagraphic Extreme Adaptive Optics (SCExAO) system. In a correlation analysis of our data sets (which use the broadband mode covering the J band through the K band in a single spectrum), we find that chromaticity in the SCExAO/CHARIS system is generally worse than temporal stability. We also develop a point-spread function (PSF) subtraction pipeline optimized for the CHARIS broadband mode, including a forward modeling-based exoplanet algorithmic throughput correction scheme. We then present contrast curves using this newly developed pipeline. An analogous subtraction of the same data sets using only the H-band slices yields the same final contrasts as the full JHK sequences; this result is consistent with our chromaticity analysis, illustrating that PSF subtraction using spectral differential imaging (SDI) in this broadband mode is generally not more effective than SDI in the individual J, H, or K bands. In the future, the data processing framework and analysis developed in this paper will be important to consider for additional SCExAO/CHARIS broadband observations and other ExAO instruments which plan to implement a similar integral field spectrograph broadband mode.

Benjamin L. Gerard↗
NASA NTRS2015

Supercomputing in the Age of Discovering Superearths, Earths and Exoplanet Systems

With the Kepler Mission, exoplanet science has entered the era of Big Data and has benefited tremendously from the availability of the NAS (NASA Advanced Supercomputing Division) Pleiades supercomputer. This presentation provides an overview of the history of exoplanet discoveries and highlights the key enabling technologies for Kepler. I provide an overview of the Kepler science pipeline and the algorithms used to analyze the data and detect transiting planets, all of which run routinely on the NAS Pleiades system. The presentation also provides a description of the Blender analysis program used to validate Kepler planet discoveries statistically, and the effort to discover exomoons lurking in Kepler data, both of which also run on Pleiades. Finally, I describe NASA's TESS (Transiting Exoplanet Survey Satellite) Mission, which leverages Kepler heritage to provide a science pipeline similar to that for Kepler and that will also run routinely on the NAS Pleiades supercomputer.

big data↗
NASA NTRS2018

Geonex: A NASA-NOAA Collaboration for Producing Land Surface Products from Geostationary Sensors Using Cloud Computing

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GEONEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GEONEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement.Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GEONEX pipeline including choice CPUs, storage media, and automation. We estimate the cost of deploying GEONEX to be $400 - 750 a month for processing data (every 30 minutes) and producing products over the conterminous US. For products such as Fire, latency can be as little as 10 minutes from the time of data acquisition.

Geostationary↗