Search NASA⌕ Search

SEARCH · Search NASA

Results for “pipeline data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Updates in Developing a Prototype Science Pipeline and Full-Volume, Global Hyperspectral Synthetic Data Sets for NASA’s Earth System Observatory’s Upcoming Surface, Biology and Geology Mission

The Surface Biology and Geology (SBG) mission recently passed mission confirmation review and has entered phase A – design and development. SBG will acquire high resolution solar-reflected spectroscopy and thermal infrared observations at a data rate of ~2.5 TB/day and generate products at ~40 TB/day. Given that the per-day volume is greater than NASA’s total extant airborne hyperspectral data collection, collecting, processing, disseminating, and exploiting the SBG data present new challenges. To meet these challenges, we have developed a prototype science pipeline and a full-volume global hyperspectral synthetic data set to help prepare for SBG’s flight (see poster GC42D-0730). Our science pipeline is based on the science processing technology developed for NASA’s Kepler and TESS planet-hunting missions. The pipeline infrastructure, Ziggy, provides a scalable architecture for robust, repeatable, and replicable science and application products that can be run on a range of systems from a laptop to the cloud or a supercomputer. Ziggy is compliant with NASA Procedural Requirement (NPR) 7150.2C, is at a technical readiness level (TRL) of 7 and has been released to github.com/nasa/ziggy. We integrated Ziggy with EO-1/Hyperion workflows to build a prototype pipeline and ingested the 17-year mission archive that provides globally sampled visible through shortwave infrared spectra that are representative of SBG data types and volumes. We fully implemented the first stage and processed the entire 55 TB Hyperion data set from the raw data (Level 0) to top-of-the-atmosphere radiance (Level 1R). We are currently evaluating the ISOFIT atmospheric correction module to convert the L1R data to surface reflectance (Level 2) before reprocessing the full data set to L2. Crosschecks are being performed with RadCalNet as well as with coincident observations by AVIRIS. We are also investigating modern methods for georectifying the Hyperion scenes. Finally, we describe an analysis of the cost to conduct forward processing and reprocessing campaigns for SBG on HECC with dedicated compute and storage resources using the resurrected Hyperion pipeline as a proxy for full-volume SBG data. The analysis demonstrates that SBG L0 data can be processed to L2 on HECC with full reprocessing campaigns every two years for ~$2.6M over a 7-year lifespan. Moreover, 69% of the system capacity would be available for other activities, possibly enabling future open-source science activities, including algorithm development, L3+ processing, .etc.

ESD↗
NASA NTRS2009

Hardware Implementation of a Bilateral Subtraction Filter

A bilateral subtraction filter has been implemented as a hardware module in the form of a field-programmable gate array (FPGA). In general, a bilateral subtraction filter is a key subsystem of a high-quality stereoscopic machine vision system that utilizes images that are large and/or dense. Bilateral subtraction filters have been implemented in software on general-purpose computers, but the processing speeds attainable in this way even on computers containing the fastest processors are insufficient for real-time applications. The present FPGA bilateral subtraction filter is intended to accelerate processing to real-time speed and to be a prototype of a link in a stereoscopic-machine- vision processing chain, now under development, that would process large and/or dense images in real time and would be implemented in an FPGA. In terms that are necessarily oversimplified for the sake of brevity, a bilateral subtraction filter is a smoothing, edge-preserving filter for suppressing low-frequency noise. The filter operation amounts to replacing the value for each pixel with a weighted average of the values of that pixel and the neighboring pixels in a predefined neighborhood or window (e.g., a 9 9 window). The filter weights depend partly on pixel values and partly on the window size. The present FPGA implementation of a bilateral subtraction filter utilizes a 9 9 window. This implementation was designed to take advantage of the ability to do many of the component computations in parallel pipelines to enable processing of image data at the rate at which they are generated. The filter can be considered to be divided into the following parts (see figure): a) An image pixel pipeline with a 9 9- pixel window generator, b) An array of processing elements; c) An adder tree; d) A smoothing-and-delaying unit; and e) A subtraction unit. After each 9 9 window is created, the affected pixel data are fed to the processing elements. Each processing element is fed the pixel value for its position in the window as well as the pixel value for the central pixel of the window. The absolute difference between these two pixel values is calculated and used as an address in a lookup table. Each processing element has a lookup table, unique for its position in the window, containing the weight coefficients for the Gaussian function for that position. The pixel value is multiplied by the weight, and the outputs of the processing element are the weight and pixel-value weight product. The products and weights are fed to the adder tree. The sum of the products and the sum of the weights are fed to the divider, which computes the sum of products the sum of weights. The output of the divider is denoted the bilateral smoothed image. The smoothing function is a simple weighted average computed over a 3 3 subwindow centered in the 9 9 window. After smoothing, the image is delayed by an additional amount of time needed to match the processing time for computing the bilateral smoothed image. The bilateral smoothed image is then subtracted from the 3 3 smoothed image to produce the final output. The prototype filter as implemented in a commercially available FPGA processes one pixel per clock cycle. Operation at a clock speed of 66 MHz has been demonstrated, and results of a static timing analysis have been interpreted as suggesting that the clock speed could be increased to as much as 100 MHz.

Huertas, Andres↗
NASA NTRS2012

Vehicle Detection for RCTA/ANS (Autonomous Navigation System)

Using a stereo camera pair, imagery is acquired and processed through the JPLV stereo processing pipeline. From this stereo data, large 3D blobs are found. These blobs are then described and classified by their shape to determine which are vehicles and which are not. Prior vehicle detection algorithms are either targeted to specific domains, such as following lead cars, or are intensity- based methods that involve learning typical vehicle appearances from a large corpus of training data. In order to detect vehicles, the JPL Vehicle Detection (JVD) algorithm goes through the following steps: 1. Take as input a left disparity image and left rectified image from JPLV stereo. 2. Project the disparity data onto a two-dimensional Cartesian map. 3. Perform some post-processing of the map built in the previous step in order to clean it up. 4. Take the processed map and find peaks. For each peak, grow it out into a map blob. These map blobs represent large, roughly vehicle-sized objects in the scene. 5. Take these map blobs and reject those that do not meet certain criteria. Build descriptors for the ones that remain. Pass these descriptors onto a classifier, which determines if the blob is a vehicle or not. The probability of detection is the probability that if a vehicle is present in the image, is visible, and un-occluded, then it will be detected by the JVD algorithm. In order to estimate this probability, eight sequences were ground-truthed from the RCTA (Robotics Collaborative Technology Alliances) program, totaling over 4,000 frames with 15 unique vehicles. Since these vehicles were observed at varying ranges, one is able to find the probability of detection as a function of range. At the time of this reporting, the JVD algorithm was tuned to perform best at cars seen from the front, rear, or either side, and perform poorly on vehicles seen from oblique angles.

Brennan, Shane↗
NASA NTRS1981

Real-time multiple-look synthetic aperture radar processor for spacecraft applications

A spaceborne synthetic aperture radar (SAR) having pipeline multiple-look data processing is described which makes use of excessive azimuth bandwidth in radar echo signals to produce multiple-looking images. Time multiplexed single-look image lines from an azimuth correlator go through an energy analyzer which analyzes the mean energy in each separate look to determine the radar antenna electric boresight for use in generating the correct reference functions for the production of high quality SAR images. The multiplexed single look image lines also go through a registration delay to produce multi-look images.

Wu, C.↗
NASA NTRS1983

High-speed digital signal normalization for feature identification

A design approach for high speed normalization of digital signals was developed. A reciprocal look up table technique is employed, where a digital value is mapped to its reciprocal via a high speed memory. This reciprocal is then multiplied with an input signal to obtain the normalized result. Normalization improves considerably the accuracy of certain feature identification algorithms. By using the concept of pipelining the multispectral sensor data processing rate is limited only by the speed of the multiplier. The breadboard system was found to operate at an execution rate of five million normalizations per second. This design features high precision, a reduced hardware complexity, high flexibility, and expandability which are very important considerations for spaceborne applications. It also accomplishes a high speed normalization rate essential for real time data processing.

Ortiz, J. A.↗
NASA NTRSDate not supplied

NASA’s Unsteady Pressure-Sensitive Paint Research and Operational Capability Developments

To address challenges in the field of unsteady aerodynamics, NASA has developed a new state-of-the-art capability called Unsteady Pressure-Sensitive Paint (uPSP). It has been developed as an operational surface-pressure measurement capability for deployment in NASA’s AETC ground test facilities. Offering unprecedented spatiotemporal resolution, uPSP is an extremely powerful tool for investigating unsteady separated flows. To accelerate scientific discovery, uPSP data is to be processed and distributed as rapidly as it is acquired. Several demonstrations of the technology collected valuable data that has been used to develop data products, a robust processing pipeline, and other computational capabilities. A collection of papers documents the most recent research and development work on the uPSP technology and capability. This paper summarizes the current state of this effort at NASA.

Pressure-Sensitive Paint↗
NASA NTRSDate not supplied

NASA’s Unsteady Pressure-Sensitive Paint Research and Operational Capability Developments

To address challenges in the field of unsteady aerodynamics, NASA has developed a new stateof-the-art capability called Unsteady Pressure-Sensitive Paint (uPSP). It has been developed as an operational surface-pressure measurement capability for deployment in NASA’s AETC ground test facilities. Offering unprecedented spatiotemporal resolution, uPSP is an extremely powerful tool for investigating unsteady separated flows. To accelerate scientific discovery, uPSP data is to be processed and distributed as rapidly as it is acquired. Several demonstrations of the technology collected valuable data that has been used to develop data products, a robust processing pipeline, and other computational capabilities. A collection of papers documents the most recent research and development work on the uPSP technology and capability. This paper summarizes the current state of this effort at NASA.

Pressure-Sensitive Paint↗
NASA NTRS2020

TESS Data Release Notes:Reprocessing of Sectors 14–19, DR30 & DR33

TESS data release 30 (DR30) provides reprocessed data products of Sector 14 to 19. The updated data products were generated using version 4.0 of the science processing pipeline and conform to the final set of data anomaly flags defined over the last two years of TESS data analysis and pipeline development. Data release 33 (DR33) corresponds to a multisector search for transiting planets in the same reprocessed data. A detailed description of the changes in the data products in DR30 and DR33 is discussed in§2, and a brief list of changes is summarized here: The timestamps for 2 minute cadence and FFI data are more accurate. The differences between reprocessed data and previous data releases are less than 2.0 seconds in all cases. Photometric apertures were increased in size for targets with T mag<11. Three new Data Anomaly Flags were added to mitigate the effects of scattered light:–Cadences with strong scattered light signals or saturation effects that corrupt the calibration data are flagged and removed from analysis (bit 15, value 16384, “Bad Calibration Exclude”).–Scattered light data anomaly flags are customized for each target, and flagged automatically based on the local background level (bit 13, value 4096, ”Scattered light flag”).–Cadences with insufficient targets to derive cotrending basis vectors are flagged and the PDCSAP FLUX light curves are set to NULL at these times (bit 16, value 32768, “Insufficient Targets for Error Correction Exclude”). The planet search of the reprocessed light curves produced a different set of TCEs from the original processed data. Although there is a high degree of overlap between the original and reprocessed data (∼83% of targets produced TCEs in common), new TCEs were produced in DR30 and not every TCE from previous data releases was recovered. The same is true of the multisector search results from DR33 compared to DR28

TESS'↗
NASA NTRS2001

Processors, Pipelines, and Protocols for Advanced Modeling Networks

Predictive capabilities arise from our understanding of natural processes and our ability to construct models that accurately reproduce these processes. Although our modeling state-of-the-art is primarily limited by existing computational capabilities, other technical areas will soon present obstacles to the development and deployment of future predictive capabilities. Advancement of our modeling capabilities will require not only faster processors, but new processing algorithms, high-speed data pipelines, and a common software engineering framework that allows networking of diverse models that represent the many components of Earth's climate and weather system. Development and integration of these new capabilities will pose serious challenges to the Information Systems (IS) technology community. Designers of future IS infrastructures must deal with issues that include performance, reliability, interoperability, portability of data and software, and ultimately, the full integration of various ES model systems into a unified ES modeling network.

Coughlan, Joseph↗
NASA NTRS2023

BatAnalysis - A Comprehensive Python Pipeline for Swift BAT Survey Analysis

The Swift Burst Alert Telescope (BAT) is a coded aperture gamma-ray instrument with a large field of view that primarily operates in survey mode when it is not triggering on transient events. The survey data consists of eighty-channel detector plane histograms that accumulate photon counts over time periods of at least 5 minutes. These histograms are processed on the ground and are used to produce the survey dataset between 14 and 195 keV. Survey data comprises >90% of all BAT data by volume and allows for the tracking of long term light curves and spectral properties of cataloged and uncataloged hard X-ray sources. Until now, the survey dataset has not been used to its full potential due to the complexity associated with its analysis and the lack of easily usable pipelines. Here, we introduce the BatAnalysis python package , a wrapper for HEASoftpy, which provides a modern, open-source pipeline to process and analyze BAT survey data. BatAnalysis allows members of the community to use BAT survey data in more advanced analyses of astrophysical sources including pulsars, pulsar wind nebula, active galactic nuclei, and other known/unknown transient events that may be detected in the hard X-ray band. We outline the steps taken by the python code and exemplify its usefulness and accuracy by analyzing survey data from the Crab Pulsar, NGC 2992, and a previously uncataloged MAXI Transient. The BatAnalysis package allows for ∼ 18 years of BAT survey to be used in a systematic way to study a large variety of astrophysical sources.

Tyler Parsotan↗
NASA NTRS1988

Real time pipelined system for forming the sum of products in the processing of video data

A 3-by-3 convolver utilizes 9 binary arithmetic units connected in cascade for multiplying 12-bit binary pixel values P sub i which are positive or two's complement binary numbers by 5-bit magnitide (plus sign) weights W sub i which may be positive or negative. The weights are stored in registers including the sign bits. For a negative weight, the one's complement of the pixel value to be multiplied is formed at each unit by a bank of 17 exclusive or gates G sub i under control of the sign of the corresponding weight W sub i, and a correction is made by adding the sum of the absolute values of all the negative weights for each 3-by-3 kernel. Since this correction value remains constant as long as the weights are constant, it can be precomputed and stored in a register as a value to be added to the product PW of the first arithmetic unit.

Wilcox, Brian↗
NASA NTRS2010

Pixel-Level Calibration in the Kepler Science Operations Center Pipeline

We present an overview of the pixel-level calibration of flight data from the Kepler Mission performed within the Kepler Science Operations Center Science Processing Pipeline. This article describes the calibration (CAL) module, which operates on original spacecraft data to remove instrument effects and other artifacts that pollute the data. Traditional CCD data reduction is performed (removal of instrument/detector effects such as bias and dark current), in addition to pixel-level calibration (correcting for cosmic rays and variations in pixel sensitivity), Kepler-specific corrections (removing smear signals which result from the lack of a shutter on the photometer and correcting for distortions induced by the readout electronics), and additional operations that are needed due to the complexity and large volume of flight data. CAL operates on long (~30 min) and short (~1 min) sampled data, as well as full-frame images, and produces calibrated pixel flux time series, uncertainties, and other metrics that are used in subsequent Pipeline modules. The raw and calibrated data are also archived in the Multi-mission Archive at Space Telescope at the Space Telescope Science Institute for use by the astronomical community.

Calibration↗
NASA NTRS2004

Searching for Pulsations from CEN X-4

The goal of this proposal is to detect or place sensitive upper limits on pulsations from the quiescent neutron-star X-ray binary Cen X-4. The observations were obtained on 1 March 2003. Unfortunately, due to problems with the standard pipeline processing of this observation, the data were not delivered until more than one year later, on 21 April 2004. No pulsations have yet been found in the data set. However, new optical spectra have been obtained which will lead to better constraints on the orbital period of Cen X-4 and which may enable detection of pulsations.

Kaaret, Philip↗
NASA NTRS2014

The Spitzer Survey of Interstellar Clouds in the Gould Belt. VI. The Auriga-California Molecular Cloud Observed with IRAC and MIPS

We present observations of the Auriga-California Molecular Cloud (AMC) at 3.6, 4.5, 5.8, 8.0, 24, 70 and 160 micrometers observed with the IRAC and MIPS detectors as part of the Spitzer Gould Belt Legacy Survey. The total mapped areas are 2.5 deg(exp 2) with IRAC and 10.47 deg2 with MIPS. This giant molecular cloud is one of two in the nearby Gould Belt of star-forming regions, the other being the Orion A Molecular Cloud (OMC). We compare source counts, colors and magnitudes in our observed region to a subset of the SWIRE data that was processed through our pipeline. Using color-magnitude and color-color diagrams, we find evidence for a substantial population of 166 young stellar objects (YSOs) in the cloud, many of which were previously unknown. Most of this population is concentrated around the LkH(alpha) 101 cluster and the filament extending from it. We present a quantitative description of the degree of clustering and discuss the fraction of YSOs in the region with disks relative to an estimate of the diskless YSO population. Although the AMC is similar in mass, size and distance to the OMC, it is forming about 15 - 20 times fewer stars.

Orion A Molecular Cloud (OMC)↗
NASA NTRS2012

The HEASARC Swift Gamma-Ray Burst Archive: The Pipeline and the Catalog

Since its launch in late 2004, the Swift satellite triggered or observed an average of one gamma-ray burst (GRB) every 3 days, for a total of 771 GRBs by 2012 January. Here, we report the development of a pipeline that semi automatically performs the data-reduction and data-analysis processes for the three instruments on board Swift (BAT, XRT, UVOT). The pipeline is written in Perl, and it uses only HEAsoft tools and can be used to perform the analysis of a majority of the point-like objects (e.g., GRBs, active galactic nuclei, pulsars) observed by Swift. We run the pipeline on the GRBs, and we present a database containing the screened data, the output products, and the results of our ongoing analysis. Furthermore, we created a catalog summarizing some GRB information, collected either by running the pipeline or from the literature. The Perl script, the database, and the catalog are available for downloading and querying at the HEASARC Web site.

BURST ARCHIVE↗
NASA NTRS2000

A Complete UV Atlas of Standard Stars

The general objective of this project is to provide a comprehensive ultraviolet spectral atlas of stars based on the data in the IUE Final Archive. The data have been uniformly processed by the IUE NEWSIPS pipeline system. The NEWSIPS gives an improved signal to noise ratio for the data that will be especially beneficial to low signal portions of the spectra; and it also gives more accurate fluxes for stars by adopting the latest absolute calibration and measured instrument parameters. Our new atlas, therefore, provides more uniform and accurate data than the version previously published. In addition, considerable efforts were devoted to obtain reliable spectral types, V,B-V, and E(B-V). Our goal is to provide the information which can be used with reasonable confidence by scientists.

Wu, Chi-Chao↗
NASA NTRS2021

I’ll have the Porter: Interactively Visualizing the Results of Statistical Maneuver Analysis

Mission design and navigation relies on statistical maneuver analysis and Monte Carlo simulations when evaluating candidate mission trajectories. Engineers must analyze large quantities of data to optimize mission safety and propellant margins, but currently rely on static text files and dense PDF slide decks to review simulation results. This approach is time-intensive, non-interactive, and difficult to share or coordinate with other mission designers. To improve this process, we present Porter: a web-based interactive mission analysis tool. This work describes Porter’s user-centered design process, its processing pipeline for importing and processing LAMBIC simulation data, core interactive features, and preliminary user feedback.

Ramaswamy, Basak Alper↗
NASA NTRSDate not supplied

Synthetic Hyperspectral Data for Global Water Quality Algorithm Development

Eutrophication and increasing prevalence of potentially toxic algal blooms (cyanoHABs) among global inland water bodies have become a major ecological concern and require direct attention. There is now a growing necessity to develop pragmatic approaches that allow timely and effective extrapolation of local aquatic processes, to spatially resolved global products. Planned aquatic biogeochemistry remote sensing data products from hyperspectral imagers such as NASA’s Surface Biology and Geology (SBG) mission and relevant aquatic sensor sensitivity precursor airborne imaging spectrometer data provide unprecedented radiometric resolution and sensor sensitivity for characterizing complex aquatic ecosystems. However, scarcity of high-quality freshwater in-situ optical data hinders our capability to develop and validate robust retrieval algorithms. A state-of-the-art synthetic dataset of paired top-of-atmosphere, bottom-of-atmosphere, and optical and biogeophysical data was developed through radiative transfer modeling to simulate natural freshwater ecosystems. A synthetic or precursor dataset for SBG is being used to train robust machine learning models to derive water quality products pertinent to SBG mission objectives. The dataset is also used to show the potential of performing vigorous aquatic sensitivity studies and explored pathways for how best to optimize hyperspectral data for machine learning development. A processing pipeline and resultant global synthetic/precursor dataset for inland waters is presented to establish the innovation for water quality studies of inland waters globally. Optical Society of America Imaging and Applied Optics Congress, Hyperspectral Imaging and Sounding of the Environment (OSA HISE) Meeting, 19-23 July 2021, Virtual Meeting, https://www.osa.org/enus/meetings/osa_meetings/optical_sensors_and_sensing_congress/program/hyperspectral_imaging_and_sounding_of_the_environm/

Synthetic↗