SEARCH · Search NASA

Results for “pipeline data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

All sources NASA NTRS DOE OSTI

NASA NTRS1977

Data handling for the geometric correction of large images

Several geometric distortions are present in remotely sensed images depending on the type of sensors and the object being observed. It is often desirable to compensate for these distortions and store the images in reference to a standard coordinate system. Digital techniques for correction are versatile and introduce a minimum of radiometric errors. The main problems to be considered in this area are the determination of the corrective transformation, resampling, and the management of the large quantities of data. It is shown that, by a judicious rearrangement of the input data, considerable reductions in the required memory capacity can be achieved. The rearrangement can be accomplished in several stages. The method presented here is amenable to pipeline implementation for processing a continuous stream of images.

Ramapriyan, H. K.↗

NASA NTRS1986

Integrated-Circuit Active Digital Filter

Pipeline architecture with parallel multipliers and adders speeds calculation of weighted sums. Picture-element values and partial sums flow through delay-adder modules. After each cycle or time unit of calculation, each value in filter moves one position right. Digital integrated-circuit chips with pipeline architecture rapidly move 35 X 35 two-dimensional convolutions. Need for such circuits in image enhancement, data filtering, correlation, pattern extraction, and synthetic-aperture-radar image processing: all require repeated calculations of weighted sums of values from images or two-dimensional arrays of data.

Nathan, R.↗

NASA NTRS1994

Feasibility study, software design, layout and simulation of a two-dimensional Fast Fourier Transform machine for use in optical array interferometry

The goal of this project was the feasibility study of a particular architecture of a digital signal processing machine operating in real time which could do in a pipeline fashion the computation of the fast Fourier transform (FFT) of a time-domain sampled complex digital data stream. The particular architecture makes use of simple identical processors (called inner product processors) in a linear organization called a systolic array. Through computer simulation the new architecture to compute the FFT with systolic arrays was proved to be viable, and computed the FFT correctly and with the predicted particulars of operation. Integrated circuits to compute the operations expected of the vital node of the systolic architecture were proven feasible, and even with a 2 micron VLSI technology can execute the required operations in the required time. Actual construction of the integrated circuits was successful in one variant (fixed point) and unsuccessful in the other (floating point).

Boriakoff, Valentin↗

NASA NTRS2002

A Proposal to Investigate Outstanding Problems in Astronomy

During the period leading up to the spectacular launch of the Space Shuttle Columbia (STS-109) on 1 March 2002 6:22 am EST, the team worked hard on a myriad of tasks to be ready for launch. Our launch support included preparations and rehearsals for the support during the mission, preparation for the SMOV and ERO program, and work to have the science team's data pipeline (APSIS) and data archive (SDA) ready by launch. A core of the team that was at the GSFC during the EVA that installed ACS monitored the turn-on and aliveness tests of ACS. One hour after installation of ACS in the HST George Hartig was showing those of us at Goddard the telemetry which demonstrated that the HRC and WFC CCDs were cooling to their preset temperatures. The TECs had survived launch! After launch, the team had several immediate and demanding tasks. We had to process the ERO observations through our pipeline and understand the limitations of the ground based-based calibrations, and simultaneously prepare the EROs for public release. The ERO images and the SMOV calibrations demonstrated that ACS met or exceeded its specifications for image quality and sensitivity. It is the most sensitive instrument that Hubble has had. The ERO images themselves made the front page of all of the major newspapers in the US. During the months after launch we have worked on the SMOV observations, and are analyzing the data from our science program.

Ford, Holland↗

NASA NTRS2016

Using Docker Containers to Extend Reproducibility Architecture for the NASA Earth Exchange (NEX)

NASA Earth Exchange (NEX) is a data, supercomputing and knowledge collaboratory that houses NASA satellite, climate and ancillary data where a focused community can come together to address large-scale challenges in Earth sciences. As NEX has been growing into a petabyte-size platform for analysis, experiments and data production, it has been increasingly important to enable users to easily retrace their steps, identify what datasets were produced by which process chains, and give them ability to readily reproduce their results. This can be a tedious and difficult task even for a small project, but is almost impossible on large processing pipelines. We have developed an initial reproducibility and knowledge capture solution for the NEX, however, if users want to move the code to another system, whether it is their home institution cluster, laptop or the cloud, they have to find, build and install all the required dependencies that would run their code. This can be a very tedious and tricky process and is a big impediment to moving code to data and reproducibility outside the original system. The NEX team has tried to assist users who wanted to move their code into OpenNEX on Amazon cloud by creating custom virtual machines with all the software and dependencies installed, but this, while solving some of the issues, creates a new bottleneck that requires the NEX team to be involved with any new request, updates to virtual machines and general maintenance support. In this presentation, we will describe a solution that integrates NEX and Docker to bridge the gap in code-to-data migration. The core of the solution is saemi-automatic conversion of science codes, tools and services that are already tracked and described in the NEX provenance system, to Docker - an open-source Linux container software. Docker is available on most computer platforms, easy to install and capable of seamlessly creating and/or executing any application packaged in the appropriate format. We believe this is an important step towards seamless process deployment in heterogeneous environments that will enhance community access to NASA data and tools in a scalable way, promote software reuse, and improve reproducibility of scientific results.

earth exchange↗

NASA NTRS2008

NASA Tech Briefs, November 2008

Topics covered include: Digital Phase Meter for a Laser Heterodyne Interferometer; Vision System Measures Motions of Robot and External Objects; Advanced Precipitation Radar Antenna to Measure Rainfall From Space; Wide-Band Radar for Measuring Thickness of Sea Ice; Vertical Isolation for Photodiodes in CMOS Imagers; Wide-Band Microwave Receivers Using Photonic Processing; L-Band Transmit/Receive Module for Phase-Stable Array Antennas; Microwave Power Combiner/Switch Utilizing a Faraday Rotator; Compact Low-Loss Planar Magic-T; Using Pipelined XNOR Logic to Reduce SEU Risks in State Machines; Quasi-Optical Transmission Line for 94-GHz Radar; Next Generation Flight Controller Trainer System; Converting from DDOR SASF to APF; Converting from CVF to AAF; Documenting AUTOGEN and APGEN Model Files; Sequence History Update Tool; Extraction and Analysis of Display Data; MRO DKF Post-Processing Tool; Rig Diagnostic Tools; MRO Sequence Checking Tool; Science Activity Planner for the MER Mission; UAVSAR Flight-Planning System; Templates for Deposition of Microscopic Pointed Structures; Adjustable Membrane Mirrors Incorporating G-Elastomers; Hall-Effect Thruster Utilizing Bismuth as Propellant; High-Temperature Crystal-Growth Cartridge Tubes Made by VPS; Quench Crucibles Reinforced with Metal; Deep-Sea Hydrothermal-Vent Sampler; Mars Rocket Propulsion System; Two-Stage Passive Vibration Isolator; Improved Thermal Design of a Compression Mold; Enhanced Pseudo-Waypoint Guidance for Spacecraft Maneuvers; Altimetry Using GPS-Reflection/Occultation Interferometry; Thermally Driven Josephson Effect; Perturbation Effects on a Supercritical C7H16/N2 Mixing Layer; Gold Nanoparticle Labels Amplify Ellipsometric Signals; Phase Matching of Diverse Modes in a WGM Resonator; WGM Resonators for Terahertz-to-Optical Frequency Conversion; Determining Concentration of Nanoparticles from Ellipsometry; Microwave-to-Optical Conversion in WGM Resonators; Four-Pass Coupler for Laser-Diode-Pumped Solid-State Laser; Low-Resolution Raman-Spectroscopy Combustion Thermometry; Temperature Sensors Based on WGM Optical Resonators; Varying the Divergence of Multiple Parallel Laser Beams; Efficient Algorithm for Rectangular Spiral Search; Algorithm-Based Fault Tolerance Integrated with Replication; Targeting and Localization for Mars Rover Operations; Terrain-Adaptive Navigation Architecture; Self-Adjusting Hash Tables for Embedded Flight Applications; Schema for Spacecraft-Command Dictionary; Combined GMSK Communications and PN Ranging; System-Level Integration of Mass Memory; Network-Attached Solid-State Recorder Architecture; Method of Cross-Linking Aerogels Using a One-Pot Reaction Scheme; An Efficient Reachability Analysis Algorithm.

Source record↗

NASA NTRS2001

ROSAT Science Data Center

This report provides a summary of the Smithsonian Astrophysical Observatory (SAO) ROSAT SCIENCE DATA CENTER (RSDC) activities for the recent years of our contract. Details have already been reported in the monthly reports. The SAO was responsible for the High Resolution Imager (HRI) detector on ROSAT. We also provided and supported the HRI standard analysis software used in the pipeline processing (SASS). Working with our colleagues at the Max Planck in Garching Germany (MPE), we fixed bugs and provided enhancements. The last major effort in this area was the port from VMS/VAX to VMS/ALPHA architecture. In 1998, a timing bug was found in the HRI standard processing system which degraded the positional accuracy because events accessed incorrect aspect solutions. The bug was fixed and we developed off-line correction routines and provided them to the community. The Post Reduction Off-line Software (PROS) package was developed by SAO and runs in the IRAF environment. Although in recent years PROS was not a contractual responsibility of the RSDC, we continued to maintain the system and provided new capabilities such as the ability to deal with simulated AXAF data in preparation for the NASA call for proposals for Chandra. Our most recent activities in this area included the debugging necessary for newer versions of IRAF which broke some of our software. At SAO we have an operating version of PROS and hope to release a patch even though almost all functionality that was lost was subsequently recovered via an IRAF patch (i.e. most of our problems were caused by an IRAF bug).

Murray, Stephen↗

NASA NTRS2018

Generalizing a Data Analysis Pipeline in the Cloud to Handle Diverse Use Cases in NASA's EOSDIS

NASA's Earth Observing System Data and Information System (EOSDIS) is tasked with archiving and distributing Earth Observation data across a range of disciplines, including atmospheric science, oceanography, land processes, natural hazards, solar radiance and even socioeconomic aspects relating to the environment. Driven by rapidly rising data volumes, EOSDIS is migrating to a cloud computing based archive over the next few years. Although this simplifies data management somewhat, the main aim is to provide the data in an environment where end users can bring their analysis to the data rather than attempting to download and manage ever-increasing volumes. To that end, a cloud-based analysis platform is being constructed to enable data transformations, analyses and visualization without egressing the data from the cloud. In this endeavor, we expect a wide variety of users, algorithms and use cases. Consequently, the architecture of this cloud analytics platform is expressly designed to be based on open services, thus fostering an ecosystem that enables the efficient combination of common components with data-specific or analysis-specific components. Reviewed and approved by Andrew Mitchell, ESDIS project manager.

Cloud computing↗

NASA NTRS2020

Automated Cardiovascular Pathology Assessment using Semantic Segmentation and Ensemble Learning

Cardiac magnetic resonance imaging provides high spatial resolution, enabling improved extraction of important functional and morphological features for cardiovascular disease staging. Segmentation of ventricular cavities and myocardium in cardiac cine sequencing provides a basis to quantify cardiac measures such as ejection fraction. A method is presented that curtails the expense and observer bias of manual cardiac evaluation by combining semantic segmentation and disease classification into a fully automatic processing pipeline. The initial processing element consists of a robust dilated convolutional neural network architecture for voxel-wise segmentation of the myocardium and ventricular cavities. The resulting comprehensive volumetric feature matrix captures diagnostic clinical procedure data and is utilized by the final processing element to model a cardiac pathology classifier. Our approach evaluated anonymized cardiac images from a training data set of 100 patients (4 pathology groups, 1 healthy group, 20 patients per group) examined at the University Hospital of Dijon. The top average Dice index scores achieved were 0.940, 0.886, 0.849 for structure segmentation of the left ventricle (LV), myocardium and right ventricle (RV) respectively. A 5-ary pathology classification accuracy of 90% was recorded on an independent test set using the trained model. Performance results demonstrate potential for advanced machine learning methods to deliver accurate, efficient and reproducible cardiac pathological assessment.

Semantic Segmentation↗

NASA NTRS2019

Numerical Simulation on Liquid Hydrogen Chill-Down Process of Vertical Pipeline

In order to improve the cryogenic propellant management technologies for a liquid hydrogen rocket with high specific impulse, JAXA, the University of Tokyo, and the NASA Glenn Research Center have jointly organized a multi-agency model validation collaboration project. As part of this project, JAXA's boiling simulation was validated with NASA's experimental data on vertical pipeline chill-down. Simulation results were in good agreement with the experimental data obtained using an improved boiling model to reproduce the spray flow. This activity achieved liquid hydrogen turbo-pump simulation at JAXA for grasping the boiling flow phenomenon from engine cut-off to re-ignition. This joint research resulted in an international cooperative relationship for discussing the cryogenic propellant management technologies necessary to develop next-generation liquid rockets.

Umemura, Yutaka↗

NASA NTRS2019

Citizen Science Twitter Data Management for Earth Science Applications

Social media data can provide useful real-time and historical information relating to the natural world, but managing this data poses challenges. Scientists at GES DISC are exploring the potential of Twitter data to augment precipitation data from the Global Precipitation Measurement (GPM) mission. However, the format of Twitter data is unconventional in the context of NASA data centers, resulting in frustration for scientists who need to work with the data. This study investigated procedures and standards needed to properly manage Twitter data to make them compatible with these data centers. After comparing databases, the study found that the MongoDB database was best suited for the storage of raw Twitter data due to its flexibility, ability to be accessed by multiple users, and querying functionality. The study used the Python package Zarr to transform processed Twitter data into a gridded format similar to that of satellite data. Each Tweet was mapped onto a time-space grid; each grid location contained information about Tweet attributes and precipitation. The study developed a pipeline for downloading, storing, and gridding Twitter data and transformed Twitter data into an understandable format for users of NASA satellite data.

Li, Rachel↗

NASA NTRS1978

Multiprocessing tradeoffs and the wind-tunnel simulation problem

The architecture tradeoff issue is discussed, concentrating on an oversimplified version of the multiprocessing aspect. A degree of symmetric multiprocessing is unavoidable; the choice is either complete symmetric multiprocessing or a number of identical pipelines. The processing elements can be geared to do either plane-multiprocessing, or line-multiprocessing. Other important design choices are the number notation, word length, main memory size, cache memory size, and different means to implement data transport.

Chen, T. C.↗

NASA NTRS1989

Data compression for Eos on-board SAR processor

A lightweight, low-power, real-time data compressor design for the Earth Observing System (Eos) onboard synthetic aperture radar (SAR) processor is presented. The implementation is based on VLSI design of a pipelined binary tree-searched vector quantizer (VQ), utilizing space-qualifiable 1.25-micron CMOS technology. The implementation exploits VLSI system design principles such as the modularity, regular data flow, simple interconnection, localized communication, simple global control, and parallel/pipelined processing. The overall system requires 30 chips with only one VLSI processing element design. The total weight is about 1.2 lbs, with an estimated power dissipation of approximately 4 watts operating at the maximum input data rate. The projected throughput rate exceeds 5 MHz.

Chang, C. Y.↗

NASA NTRSDate not supplied

Seven Years of Bright Meteor Data from the NASA All Sky Fireball Network

The NASA All Sky Fireball Network is a network consisting of 18 all-sky meteor cameras across the continental US. We present a data release of 33,660 bright meteors collected between 2013 and 2019. The released data consists of trajectory, orbit, radiant, shower association, and brightness. We discuss the camera hardware and software used to make the observations, the characteristics of the data, and known error sources. The camera hardware consists of Watec 902H2 Ultimate CCD cameras with a fish-eye lens giving an all-sky field of view. The data is complete until an absolute meteor magnitude of approximately -4, but meteors down to a magnitude of -1 are routinely observed. The cameras are spread across the continental United States in five subnetworks, with each network consisting of two to five cameras. The cameras report to a central server where the data is processed each morning. The network is modeled after the University of Western Ontario’s Southern Ontario Meteor Network. The cameras are running All-sky and Guided Real-time Detection (ASGARD) meteor detection software and analysis pipeline.

Aaron Kingery↗

NASA NTRS2012

MER Telemetry Processor

MERTELEMPROC processes telemetered data in data product format and generates Experiment Data Records (EDRs) for many instruments (HAZCAM, NAVCAM, PANCAM, microscopic imager, Moessbauer spectrometer, APXS, RAT, and EDLCAM) on the Mars Exploration Rover (MER). If the data is compressed, then MERTELEMPROC decompresses the data with an appropriate decompression algorithm. There are two compression algorithms (ICER and LOCO) used in MER. This program fulfills a MER specific need to generate Level 1 products within a 60-second time requirement. EDRs generated by this program are used by merinverter, marscahv, marsrad, and marsjplstereo to generate higher-level products for the mission operations. MERTELEPROC was the first GDS program to process the data product. Metadata of the data product is in XML format. The software allows user-configurable input parameters, per-product processing (not streambased processing), and fail-over is allowed if the leading image header is corrupted. It is used within the MER automated pipeline. MERTELEMPROC is part of the OPGS (Operational Product Generation Subsystem) automated pipeline, which analyzes images returned by in situ spacecraft and creates level 1 products to assist in operations, science, and outreach.

Lee, Hyun H.↗

NASA NTRS2009

Multi-Mission Automated Task Invocation Subsystem

Multi-Mission Automated Task Invocation Subsystem (MATIS) is software that establishes a distributed data-processing framework for automated generation of instrument data products from a spacecraft mission. Each mission may set up a set of MATIS servers for processing its data products. MATIS embodies lessons learned in experience with prior instrument- data-product-generation software. MATIS is an event-driven workflow manager that interprets project-specific, user-defined rules for managing processes. It executes programs in response to specific events under specific conditions according to the rules. Because requirements of different missions are too diverse to be satisfied by one program, MATIS accommodates plug-in programs. MATIS is flexible in that users can control such processing parameters as how many pipelines to run and on which computing machines to run them. MATIS has a fail-safe capability. At each step, MATIS captures and retains pertinent information needed to complete the step and start the next step. In the event of a restart, this information is retrieved so that processing can be resumed appropriately. At this writing, it is planned to develop a graphical user interface (GUI) for monitoring and controlling a product generation engine in MATIS. The GUI would enable users to schedule multiple processes and manage the data products produced in the processes. Although MATIS was initially designed for instrument data product generation,

Cheng, Cecilia S.↗

NASA NTRS2023

High-Rate Delay Tolerant Networking (HDTN) User Guide Version 1.0

Delay Tolerant Networking (DTN) has been identified as a key technology to enable and facilitate the development and growth of future space networks. Classically, space communications networks are collections of disparate links that are manually managed either point-to-point or use space relays. The accelerating accessibility of space enables a new scaling of space nodes, yet both the manual management of configurations and scheduling and the lack of structure connecting links precisely prohibit scaling. This challenge gives rise to newer and larger classes of communications needs that are met by DTN, which must overcome the disconnection, disruption, latency, and mobility featured in space communications systems. DTN joins the underlying links as an overlay, and can be made to communicate over any protocol stack. The core actions of DTN are store, carry, and forward, where data are stored instead of dropped if there is no immediately available outduct. It does this by taking the DTN unit of data, bundles, and providing necessary layers to adapt these bundles to the underlying transport protocols of choice; these are called convergence layers. DTN's Bundle Protocol (BP) can then be used on top of terrestrial protocol stacks, such as TCP/IP, as well as protocols for space, such as LTP/AOS, all in the same network. For emphasis it is noted that bundles can be of essentially any size, and hence this convergence to lower layers of choice is necessary. Existing DTN implementations have operated in constrained environments with limited resources, resulting in low data speeds. However, as various technologies have advanced, data transfer rates and efficiency have advanced, which has pushed the need for a DTN implementation for ground systems and for spacecraft that is performance-oriented in order to not impose an unnecessary bottleneck. High-rate Delay Tolerant Networking (HDTN) takes advantage of modern hardware platforms to substantially reduce latency and improve throughput compared to today’s DTN operations. The HDTN implementation maintains interoperability with existing deployments of DTN that conform to IETF RFCs 4838, 5050, and 9171. At the same time, HDTN defines a new data format better suited to higher-rate operation. It defines and adopts a massively parallel pipelined and message-oriented architecture, allowing the system to scale gracefully as its resources increase. HDTN’s architecture also supports hooks to replace various processing pipeline elements with specialized hardware accelerators. This offers improved Size, Weight, and Power (SWaP) characteristics while reducing development complexity and cost.

Delay Tolerant Networking↗

NASA NTRS2024

High-Rate Delay Tolerant Networking (HDTN) User Guide Version 1.3.0

Delay Tolerant Networking↗