Search NASA⌕ Search

SEARCH · Search NASA

Results for “pipeline data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18

Reed-Solomon decoder

A Reed-Solomon decoder with dedicated hardware for five sequential algorithms was designed with overall pipelining by memory swapping between input, processing and output memories, and internal pipelining through the five algorithms. The code definition used in decoding is specified by a keyword received with each block of data so that a number of different code formats may be decoded by the same hardware.

Lahmeyer, Charles R.↗

Core Model Proposal 401: Ukraine as an independent region in GCAM

The goal of this core model proposal (CMP) is to break out Ukraine from the Europe_Eastern region. This work aims to establish Ukraine as an independent region in the GCAM core (region 14) while moving Belarus and Moldova to region 15 (Europe_Non_EU). We have: 1) Updated several mappings to recode region 14 (formerly Europe_Eastern) as Ukraine and moved Belarus and Moldova to region 15 (Europe_Non_EU); 2) Updated several assumptions in the raw data files which provide information by region to reframe Ukraine as the 14th region, including coefficients, base year values, share weight interpolation values and rules, pipeline networks for gas trade, elasticities, shares, etc. 3) Changed documentation and in-code comments at several places referring to fixed 32 regions in GCAM to indicate that GCAM can have any number of regions; 4) Updated code base in gcamdata to dynamically process data for Ukraine given special cases.

Global Change Analysis Model (GCAM)↗

OrthoPhyl—streamlining large-scale, orthology-based phylogenomic studies of bacteria at broad evolutionary scales

Abstract There are a staggering number of publicly available bacterial genome sequences (at writing, 2.0 million assemblies in NCBI's GenBank alone), and the deposition rate continues to increase. This wealth of data begs for phylogenetic analyses to place these sequences within an evolutionary context. A phylogenetic placement not only aids in taxonomic classification but informs the evolution of novel phenotypes, targets of selection, and horizontal gene transfer. Building trees from multi-gene codon alignments is a laborious task that requires bioinformatic expertise, rigorous curation of orthologs, and heavy computation. Compounding the problem is the lack of tools that can streamline these processes for building trees from large-scale genomic data. Here we present OrthoPhyl, which takes bacterial genome assemblies and reconstructs trees from whole genome codon alignments. The analysis pipeline can analyze an arbitrarily large number of input genomes (>1200 tested here) by identifying a diversity-spanning subset of assemblies and using these genomes to build gene models to infer orthologs in the full dataset. To illustrate the versatility of OrthoPhyl, we show three use cases: E. coli/Shigella, Brucella/Ochrobactrum and the order Rickettsiales. We compare trees generated with OrthoPhyl to trees generated with kSNP3 and GToTree along with published trees using alternative methods. We show that OrthoPhyl trees are consistent with other methods while incorporating more data, allowing for greater numbers of input genomes, and more flexibility of analysis.

59 BASIC BIOLOGICAL SCIENCES↗

Data handling for the geometric correction of large images

Several geometric distortions are present in remotely sensed images depending on the type of sensors and the object being observed. It is often desirable to compensate for these distortions and store the images in reference to a standard coordinate system. Digital techniques for correction are versatile and introduce a minimum of radiometric errors. The main problems to be considered in this area are the determination of the corrective transformation, resampling, and the management of the large quantities of data. It is shown that, by a judicious rearrangement of the input data, considerable reductions in the required memory capacity can be achieved. The rearrangement can be accomplished in several stages. The method presented here is amenable to pipeline implementation for processing a continuous stream of images.

Ramapriyan, H. K.↗

Integrated-Circuit Active Digital Filter

Pipeline architecture with parallel multipliers and adders speeds calculation of weighted sums. Picture-element values and partial sums flow through delay-adder modules. After each cycle or time unit of calculation, each value in filter moves one position right. Digital integrated-circuit chips with pipeline architecture rapidly move 35 X 35 two-dimensional convolutions. Need for such circuits in image enhancement, data filtering, correlation, pattern extraction, and synthetic-aperture-radar image processing: all require repeated calculations of weighted sums of values from images or two-dimensional arrays of data.

Nathan, R.↗

Feasibility study, software design, layout and simulation of a two-dimensional Fast Fourier Transform machine for use in optical array interferometry

The goal of this project was the feasibility study of a particular architecture of a digital signal processing machine operating in real time which could do in a pipeline fashion the computation of the fast Fourier transform (FFT) of a time-domain sampled complex digital data stream. The particular architecture makes use of simple identical processors (called inner product processors) in a linear organization called a systolic array. Through computer simulation the new architecture to compute the FFT with systolic arrays was proved to be viable, and computed the FFT correctly and with the predicted particulars of operation. Integrated circuits to compute the operations expected of the vital node of the systolic architecture were proven feasible, and even with a 2 micron VLSI technology can execute the required operations in the required time. Actual construction of the integrated circuits was successful in one variant (fixed point) and unsuccessful in the other (floating point).

Boriakoff, Valentin↗

An FPGA-based hardware accelerator supporting sensitive sequence homology filtering with profile hidden Markov models

Abstract Background Sequence alignment lies at the heart of genome sequence annotation. While the BLAST suite of alignment tools has long held an important role in alignment-based sequence database search, greater sensitivity is achieved through the use of profile hidden Markov models (pHMMs). Here, we describe an FPGA hardware accelerator, called HAVAC, that targets a key bottleneck step (SSV) in the analysis pipeline of the popular pHMM alignment tool, HMMER. Results The HAVAC kernel calculates the SSV matrix at 1739 GCUPS on a $$\sim$$ ∼ $3000 Xilinx Alveo U50 FPGA accelerator card, $$\sim$$ ∼ 227× faster than the optimized SSV implementation in nhmmer . Accounting for PCI-e data transfer data processing, HAVAC is 65× faster than nhmmer’s SSV with one thread and 35× faster than nhmmer with four threads, and uses $$\sim$$ ∼ 31% the energy of a traditional high end Intel CPU. Conclusions HAVAC demonstrates the potential offered by FPGA hardware accelerators to produce dramatic speed gains in sequence annotation and related bioinformatics applications. Because these computations are performed on a co-processor, the host CPU remains free to simultaneously compute other aspects of the analysis pipeline.

59 BASIC BIOLOGICAL SCIENCES↗

Protocol for applying a network-enabled gene discovery pipeline to non-model plant species

Identifying upstream regulators of key genes is essential for understanding gene regulatory mechanisms and translating these insights into functional targets. Here, we present a protocol for applying the network-enabled gene discovery pipeline (NEEDLE) to non-model plant species. We describe steps for environment setup, data preparation, computational analysis, expected outputs, and parameter considerations. NEEDLE integrates RNA sequencing (RNA-seq) processing, weighted gene co-expression analysis (WGCNA), Gene Network Inference with Ensemble of trees (GENIE3), and promoter conservation analysis to prioritize candidate transcriptional regulators.

Plant Sciences↗

A Proposal to Investigate Outstanding Problems in Astronomy

During the period leading up to the spectacular launch of the Space Shuttle Columbia (STS-109) on 1 March 2002 6:22 am EST, the team worked hard on a myriad of tasks to be ready for launch. Our launch support included preparations and rehearsals for the support during the mission, preparation for the SMOV and ERO program, and work to have the science team's data pipeline (APSIS) and data archive (SDA) ready by launch. A core of the team that was at the GSFC during the EVA that installed ACS monitored the turn-on and aliveness tests of ACS. One hour after installation of ACS in the HST George Hartig was showing those of us at Goddard the telemetry which demonstrated that the HRC and WFC CCDs were cooling to their preset temperatures. The TECs had survived launch! After launch, the team had several immediate and demanding tasks. We had to process the ERO observations through our pipeline and understand the limitations of the ground based-based calibrations, and simultaneously prepare the EROs for public release. The ERO images and the SMOV calibrations demonstrated that ACS met or exceeded its specifications for image quality and sensitivity. It is the most sensitive instrument that Hubble has had. The ERO images themselves made the front page of all of the major newspapers in the US. During the months after launch we have worked on the SMOV observations, and are analyzing the data from our science program.

Ford, Holland↗

Using Docker Containers to Extend Reproducibility Architecture for the NASA Earth Exchange (NEX)

NASA Earth Exchange (NEX) is a data, supercomputing and knowledge collaboratory that houses NASA satellite, climate and ancillary data where a focused community can come together to address large-scale challenges in Earth sciences. As NEX has been growing into a petabyte-size platform for analysis, experiments and data production, it has been increasingly important to enable users to easily retrace their steps, identify what datasets were produced by which process chains, and give them ability to readily reproduce their results. This can be a tedious and difficult task even for a small project, but is almost impossible on large processing pipelines. We have developed an initial reproducibility and knowledge capture solution for the NEX, however, if users want to move the code to another system, whether it is their home institution cluster, laptop or the cloud, they have to find, build and install all the required dependencies that would run their code. This can be a very tedious and tricky process and is a big impediment to moving code to data and reproducibility outside the original system. The NEX team has tried to assist users who wanted to move their code into OpenNEX on Amazon cloud by creating custom virtual machines with all the software and dependencies installed, but this, while solving some of the issues, creates a new bottleneck that requires the NEX team to be involved with any new request, updates to virtual machines and general maintenance support. In this presentation, we will describe a solution that integrates NEX and Docker to bridge the gap in code-to-data migration. The core of the solution is saemi-automatic conversion of science codes, tools and services that are already tracked and described in the NEX provenance system, to Docker - an open-source Linux container software. Docker is available on most computer platforms, easy to install and capable of seamlessly creating and/or executing any application packaged in the appropriate format. We believe this is an important step towards seamless process deployment in heterogeneous environments that will enhance community access to NASA data and tools in a scalable way, promote software reuse, and improve reproducibility of scientific results.

earth exchange↗

Geospatial Data Workflow Orchestration and Architecture

In an era characterized by explosive growth in geospatial data, the selection of appropriate technologies for data storage, processing, and orchestration is critical for organizations aiming to maintain competitive advantages. This white paper provides a comprehensive analysis of how Oak Ridge National Laboratory (ORNL) has effectively employed various cloud technologies, including containerized applications, container orchestrators, and workflow orchestrators, to develop robust geospatial data processing solutions. We explore the fundamental concepts behind these technologies and compare multiple deployment models tailored to diverse use cases. Our findings conclude that while Kubernetes has emerged as the preferred platform for truly scalable and fault-tolerant production workflows, the choice of workflow orchestration tool requires careful consideration of team needs, pipeline complexity, and deployment environments. This paper aims to serve as a strategic guide for organizations leveraging geospatial data, articulating the balance between technology choices and practical implementation to enhance workflow efficacy and scalability.

97 MATHEMATICS AND COMPUTING↗

Event Classifications on DNE2 Main Experiment Data using a Convolutional Neural Network Ensemble

The Dynamic Networks (DN) Experiment for FY24 (DNE2) is an experiment within DN with the goal of quantitatively evaluating the effectiveness of solutions developed so far by various researchers under the Low Yield Nuclear Monitoring (LYNM) program using a shared set of metrics and datasets. A key component of this experiment is the mimicking of a signature processing pipeline, and comparing currently accepted and standard-use processing methods to more state-of-the-art processes developed under DN. In this work, we focus specifically on the Event Characterization (EC) Focus Area (FA) of the pipeline, where a seismic event’s magnitude, yield and class are identified. We use Deep Learning (DL) to classify the type of events being processed as either earthquakes (EQs) or explosions (EXs) for three iterations of experiment datasets. The model is noticeably more confident and accurate in classifying explosions than earthquakes, reflecting a known shortcoming of the model, that being of a bias towards predicting explosions over earthquakes in the west coast due to training data biases.

97 MATHEMATICS AND COMPUTING↗

Segmentation Model Distillation [Poster]

The process of training object detection (OD) or image segmentation model requires both a substantial amount of data and technical knowledge, which often creates challenges in applying these types of models to their full potential. In order to streamline the process of developing these models, we propose a new pipeline where a foundation model assists in the dataset generation. Then this resulting dataset is used to fine-tune a fast light-weight model to perform the custom segmentation or OD. This resulting model is also fit for real-time image segmentation, such as in a video stream.

97 MATHEMATICS AND COMPUTING↗

Exploring DAOS as a Burst Buffer for a 100 Gbps DAQ Real-Time Streaming System

We present an experimental evaluation of a burst buffer for a real-time DAQ streaming system designed to transmit instrument data to remote data centers. The system is based on EJ-FAT, a load balancing system capable of Nx 100Gbps streams, distributing data from event sources to processing nodes. We explore applying the DAOS system as a burst buffer to serve a number of purposes: improve resiliency, elasticity and add new functions into the processing pipeline. In the evaluation a sender transmits events over a 100Gbps network to a receiver integrated with DAOS to store the reassembled events using DAOS APIs. We evaluate the system for possible bottlenecks and provide end-to-end evaluation with a burst buffer using DAOS storage abstractions. We show that a receiver node can support 38.1 Gbps. This proves the viability of our approach and allows us to extend this work to investigate scale-out properties and new streaming optimizations.

Mei, Xinxin↗

Advanced Distributed Optical Fiber Sensor Systems for Pipeline Integrity Monitoring

Distributed fiber optic sensors allow the measurement of structural parameters such as static/dynamic strain, temperature, pressure, and vibrations at thousands of locations along a single fiber cable. Deep neural network (DNN) algorithms were developed for rapid data processing speed and vibration event classification.

Lalam, Nageswara↗

A Radiation-Hard 8-Channel 15-Bit 40-MSPS ADC for the ATLAS Liquid Argon Calorimeter Readout

The custom design of a radiation-hardened, 8-channel, 40-MSPS, 15-bit resolution, 14.2-bit dynamic range, 11.4-ENOB ADC data acquisition ASIC fabricated in a commercial 65-nm triple-well CMOS technology is presented. The ADC is developed for and integrates seamlessly into the readout system for the ATLAS liquid argon (LAr) calorimeter in the high-luminosity large hadron collider (HLLHC) upgrade at CERN, which will require a total of 364 936 ADC channels. A three-stage MDAC+SAR pipelined ADC architecture was designed to meet the physics requirements and scientific goals of the ATLAS experiment. The ADC is a fully self-contained data acquisition system that includes foreground calibration, digital data processing, digital control, and supporting circuitry. The measured performance shows the ADC achieves a competitive dynamic range and SNDR, and it meets or exceeds the ATLAS analog requirements. Radiation tolerance and scalability design considerations were implemented at the device-, circuit-, and system-level. Radiation-hardening-by-design techniques used include redundancy for digital circuits, the use of MiM capacitors, and a hybrid RC-DAC for the ADC core. The ADC ASIC was demonstrated to be robust against the effects of the intense radiation expected in the HL-LHC experimental environment.

DAQ↗

NASA Tech Briefs, November 2008

Topics covered include: Digital Phase Meter for a Laser Heterodyne Interferometer; Vision System Measures Motions of Robot and External Objects; Advanced Precipitation Radar Antenna to Measure Rainfall From Space; Wide-Band Radar for Measuring Thickness of Sea Ice; Vertical Isolation for Photodiodes in CMOS Imagers; Wide-Band Microwave Receivers Using Photonic Processing; L-Band Transmit/Receive Module for Phase-Stable Array Antennas; Microwave Power Combiner/Switch Utilizing a Faraday Rotator; Compact Low-Loss Planar Magic-T; Using Pipelined XNOR Logic to Reduce SEU Risks in State Machines; Quasi-Optical Transmission Line for 94-GHz Radar; Next Generation Flight Controller Trainer System; Converting from DDOR SASF to APF; Converting from CVF to AAF; Documenting AUTOGEN and APGEN Model Files; Sequence History Update Tool; Extraction and Analysis of Display Data; MRO DKF Post-Processing Tool; Rig Diagnostic Tools; MRO Sequence Checking Tool; Science Activity Planner for the MER Mission; UAVSAR Flight-Planning System; Templates for Deposition of Microscopic Pointed Structures; Adjustable Membrane Mirrors Incorporating G-Elastomers; Hall-Effect Thruster Utilizing Bismuth as Propellant; High-Temperature Crystal-Growth Cartridge Tubes Made by VPS; Quench Crucibles Reinforced with Metal; Deep-Sea Hydrothermal-Vent Sampler; Mars Rocket Propulsion System; Two-Stage Passive Vibration Isolator; Improved Thermal Design of a Compression Mold; Enhanced Pseudo-Waypoint Guidance for Spacecraft Maneuvers; Altimetry Using GPS-Reflection/Occultation Interferometry; Thermally Driven Josephson Effect; Perturbation Effects on a Supercritical C7H16/N2 Mixing Layer; Gold Nanoparticle Labels Amplify Ellipsometric Signals; Phase Matching of Diverse Modes in a WGM Resonator; WGM Resonators for Terahertz-to-Optical Frequency Conversion; Determining Concentration of Nanoparticles from Ellipsometry; Microwave-to-Optical Conversion in WGM Resonators; Four-Pass Coupler for Laser-Diode-Pumped Solid-State Laser; Low-Resolution Raman-Spectroscopy Combustion Thermometry; Temperature Sensors Based on WGM Optical Resonators; Varying the Divergence of Multiple Parallel Laser Beams; Efficient Algorithm for Rectangular Spiral Search; Algorithm-Based Fault Tolerance Integrated with Replication; Targeting and Localization for Mars Rover Operations; Terrain-Adaptive Navigation Architecture; Self-Adjusting Hash Tables for Embedded Flight Applications; Schema for Spacecraft-Command Dictionary; Combined GMSK Communications and PN Ranging; System-Level Integration of Mass Memory; Network-Attached Solid-State Recorder Architecture; Method of Cross-Linking Aerogels Using a One-Pot Reaction Scheme; An Efficient Reachability Analysis Algorithm.

Source record↗

Constraining Galaxy-Halo connection using machine learning

We investigate the potential of machine learning (ML) methods to model small-scale galaxy clustering for constraining Halo Occupation Distribution (HOD) parameters. Our analysis reveals that while many ML algorithms report good statistical fits, they often yield likelihood contours that are significantly biased in both mean values and variances relative to the true model parameters. This highlights the importance of careful data processing and algorithm selection in ML applications for galaxy clustering, as even seemingly robust methods can lead to biased results if not applied correctly. ML tools offer a promising approach to exploring the HOD parameter space with significantly reduced computational costs compared to traditional brute-force methods if their robustness is established. Using our ANN-based pipeline, we successfully recreate some standard results from recent literature. Properly restricting the HOD parameter space, transforming the training data, and carefully selecting ML algorithms are essential for achieving unbiased and robust predictions. Among the methods tested, artificial neural networks (ANNs) outperform random forests (RF) and ridge regression in predicting clustering statistics, when the HOD prior space is appropriately restricted. We demonstrate these findings using the projected two-point correlation function (w p (r p )), angular multipoles of the correlation function (ξ ℓ (r)), and the void probability function (VPF) of Luminous Red Galaxies from Dark Energy Spectroscopic Instrument mocks. Our results show that while combining w p (r p ) and VPF improves parameter constraints, adding the multipoles ξ 0 , ξ 2 , and ξ 4 to w p (r p ) does not significantly improve the constraints.

cosmology↗