Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Data Release 1 of the Dark Energy Spectroscopic Instrument

In 2021 May the Dark Energy Spectroscopic Instrument (DESI) collaboration began a 5 yr spectroscopic redshift survey to produce a detailed map of the evolving three-dimensional structure of the Universe between z = 0 and z ≈ 4. DESI’s principal scientific objectives are to place precise constraints on the equation of state of dark energy, the gravitationally driven growth of large-scale structure, and the sum of the neutrino masses, and to explore the observational signatures of primordial inflation. We present DESI DR1, which consists of all data acquired during the first 13 months of the DESI main survey, as well as a uniform reprocessing of the DESI Survey Validation data, which were previously made public in the DESI Early Data Release. The DR1 main survey includes high-confidence redshifts for 18.7M objects, of which 13.1M are spectroscopically classified as galaxies, 1.6M as quasars, and 4M as stars, making DR1 the largest sample of extragalactic redshifts ever assembled. We summarize the DR1 observations, the spectroscopic data-reduction pipeline and data products, large-scale structure catalogs, value-added catalogs, and describe how to access and interact with the data. In addition to fulfilling its core cosmological objectives with unprecedented precision, we expect DR1 to enable a wide range of transformational astrophysical studies and discoveries.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

sdt (Solar Data Tools) [SWR-25-130]

Solar Data Tools (sdt) is an open-source Python library for analyzing PV power (and irradiance) time-series data. It was developed to enable analysis of unlabeled PV data, i.e. with no model, no meteorological data, and no performance index required, by taking a statistical signal processing approach in the algorithms used in the package’s main data processing pipeline. Solar Data Tools empowers PV system fleet owners or operators to analyze system performance a hundred times faster even when they only have access to the most basic data stream—power output of the system.

Meyers-Im, Bennet [National Laboratory of the Rock↗
NASA NTRS1983

Parallel processing in a host plus multiple array processor system for radar

Host plus multiple array processor architecture is demonstrated to yield a modular, fast, and cost-effective system for radar processing. Software methodology for programming such a system is developed. Parallel processing with pipelined data flow among the host, array processors, and discs is implemented. Theoretical analysis of performance is made and experimentally verified. The broad class of problems to which the architecture and methodology can be applied is indicated.

Barkan, B. Z.↗
NASA NTRS1988

VLSI high speed packet processor

The Goddard Space Flight Center Mission Operations and Data Systems Directorate has developed a packet processor card utilizing semicustom very large scale integration (VLSI) devices, microprocessors, and programmable gate arrays to support the implementation of multichannel telemetry data capture systems. This card will receive synchronized error corrected telemetry transfer frames and output annotated application packets derived from this data. An adaptable format capability is provided by the programmability of three microprocessors while the throughput capability of the packet processor is achieved by a data pipeline consisting of two separate RAM systems controlled by specially designed semicustom VLSI logic.

Grebowsky, Gerald J.↗
NASA NTRS2003

Intelligent Systems Technologies to Assist in Utilization of Earth Observation Data

With the launch of several Earth observing satellites over the last decade, we are now in a data rich environment. From NASA's Earth Observing System (EOS) satellites alone, we are accumulating more than 3 TB per day of raw data and derived geophysical parameters. The data products are being distributed to a large user community comprising scientific researchers, educators and operational government agencies. Notable progress has been made in the last decade in facilitating access to data. However, to realize the full potential of the growing archives of valuable scientific data, further progress is necessary in the transformation of data into information, and information into knowledge that can be used in particular applications. Sponsored by NASA s Intelligent Systems Project within the Computing, Information and Communication Technology (CICT) Program, a conceptual architecture study has been conducted to examine ideas to improve data utilization through the addition of intelligence into the archives in the context of an overall knowledge building system. Potential Intelligent Archive concepts include: 1) Mining archived data holdings using Intelligent Data Understanding algorithms to improve metadata to facilitate data access and usability; 2) Building intelligence about transformations on data, information, knowledge, and accompanying services involved in a scientific enterprise; 3) Recognizing the value of results, indexing and formatting them for easy access, and delivering them to concerned individuals; 4) Interacting as a cooperative node in a web of distributed systems to perform knowledge building (i.e., the transformations from data to information to knowledge) instead of just data pipelining; and 5) Being aware of other nodes in the knowledge building system, participating in open systems interfaces and protocols for virtualization, and collaborative interoperability. This paper presents some of these concepts and identifies issues to be addressed by research in future intelligent systems technology.

Ramapriyan, Hampapuram K.↗
NASA NTRS2003

A Comparison of Solar p-Mode Parameters from MDI and Gong: Mode Frequencies and Structure Inversions

Helioseismic analysis of solar global oscillations allows investigation of the internal structure of the Sun. One important test of the reliability of the inferences from helioseismology is that the results from independent sets of contemporaneous data are consistent with one another. Here we compare mode frequencies from the Global Oscillation Network Group and Michelson Doppler Imager on board SOHO and resulting inversion results on the Sun's internal structure. The average relative differences between the data sets are typically less than 1 x 10(exp -5) substantially smaller than the formal errors in the differences; however, in some cases the frequency differences show a systematic behavior that might nonetheless influence the inversion results. We find that the differences in frequencies are not a result of instrumental effects but are almost entirely related to the data pipeline software. Inversion of the frequencies shows that their differences do not result in any significant effects on the resulting inferences on solar structure. We have also experimented with fitting asymmetric profiles to the oscillation power spectra and find that, compared with the symmetric fits, this causes no significant change in the inversion results.

Basu, S.↗
NASA NTRS2012

Interactive 3D Mars Visualization

The Interactive 3D Mars Visualization system provides high-performance, immersive visualization of satellite and surface vehicle imagery of Mars. The software can be used in mission operations to provide the most accurate position information for the Mars rovers to date. When integrated into the mission data pipeline, this system allows mission planners to view the location of the rover on Mars to 0.01-meter accuracy with respect to satellite imagery, with dynamic updates to incorporate the latest position information. Given this information so early in the planning process, rover drivers are able to plan more accurate drive activities for the rover than ever before, increasing the execution of science activities significantly. Scientifically, this 3D mapping information puts all of the science analyses to date into geologic context on a daily basis instead of weeks or months, as was the norm prior to this contribution. This allows the science planners to judge the efficacy of their previously executed science observations much more efficiently, and achieve greater science return as a result. The Interactive 3D Mars surface view is a Mars terrain browsing software interface that encompasses the entire region of exploration for a Mars surface exploration mission. The view is interactive, allowing the user to pan in any direction by clicking and dragging, or to zoom in or out by scrolling the mouse or touchpad. This set currently includes tools for selecting a point of interest, and a ruler tool for displaying the distance between and positions of two points of interest. The mapping information can be harvested and shared through ubiquitous online mapping tools like Google Mars, NASA WorldWind, and Worldwide Telescope.

Powell, Mark W.↗
NASA NTRS2015

The CHARIS IFS for high contrast imaging at Subaru

The Coronagraphic High Angular Resolution Imaging Spectrograph (CHARIS) is an integral field spectrograph (IFS) being built for the Subaru telescope. CHARIS will take spectra of brown dwarfs and hot Jovian planets in the coronagraphic image provided by the Subaru Coronagraphic Extreme Adaptive Optics (SCExAO) and AO188 adaptive optics systems. The system is designed to detect objects five orders of magnitude dimmer than their parent star down to an 80 milliarcsecond inner working angle. For characterization, CHARIS has a high-resolution prism providing an average spectral resolution of R82, R69, and R82 in J, H, and K bands respectively. The so-called discovery mode uses a second low-resolution prism with an average spectral resolution of R19 spanning 1.15-2.37 microns (J+H+K bands). This is unique compared to other high contrast IFS designs. It augments low inner working angle performance by reducing the separation at which we can rely on spectral differential imaging. The principal challenge for a high-contrast IFS is quasi-static speckles, which cause undue levels of spectral crosstalk. CHARIS has addressed this through several key design aspects that should constrain crosstalk between adjacent spectral features to be below 1%. Sitting on the Nasmyth platform, the alignment between the lenslet array, prism, and detector will be highly stable, key for the performance of the data pipeline. Nearly every component has arrived and the project is entering its final build phase. Here we review the science case, the resulting design, status of final construction, and lessons learned that are directly applicable to future exoplanet instruments.

Subaru Coronagraphic Extreme Adaptive Optics (SCEx↗
NASA NTRS2016

Enabling the Direct Detection of Earth-Sized Exoplanets with the LBTI HOSTS Project: A Progress Report

NASA has funded a project called the Hunt for Observable Signatures of Terrestrial Systems (HOSTS) to survey nearby solar type stars to determine the amount of warm zodiacal dust in their habitable zones. The goal is not only to determine the luminosity distribution function but also to know which individual stars have the least amount of zodiacal dust. It is important to have this information for future missions that directly image exoplanets as this dust is the main source of astrophysical noise for them. The HOSTS project utilizes the Large Binocular Telescope Interferometer (LBTI), which consists of two 8.4-m apertures separated by a 14.4-m baseline on Mt. Graham, Arizona. The LBTI operates in a nulling mode in the mid-infrared spectral window (8-13 micrometers), in which light from the two telescopes is coherently combined with a 180 degree phase shift between them, producing a dark fringe at the location of the target star. In doing so the starlight is greatly reduced, increasing the contrast, analogous to a coronagraph operating at shorter wavelengths. The LBTI is a unique instrument, having only three warm reflections before the starlight reaches cold mirrors, giving it the best photometric sensitivity of any interferometer operating in the mid-infrared. It also has a superb Adaptive Optics (AO) system giving it Strehl ratios greater than 98% at 10 micrometers. In 2014 into early 2015 LBTI was undergoing commissioning. The HOSTS. project team passed its Operational Readiness Review (ORR) in April 2015. The team recently published papers on the target sample, modeling of the nulled disk images, and initial results such as the detection of warm dust around eta Corvi. Recently a paper was published on the data pipeline and on-sky performance. An additional paper is in preparation on Beta Leo. We will discuss the scientific and programmatic context for the LBTI project, and we will report recent progress, new results, and plans for the science verification phase that started in February 2016, and for the survey.

Hunt for Observable Signatures of Terrestrial Syst↗
NASA NTRS2018

The Gaia Catalogue Second Data Release and Its Implications to Optical Observations of Man-Made Earth Orbiting Objects

The Gaia catalogue second data release and its implications to optical observations of man-made Earth orbiting objects. Abstract and not the Final Paper is attached. The Gaia spacecraft was launched in December 2013 by the European Space Agency to produce a three-dimensional, dynamic map of objects within the Milky Way. Gaia's first year of data was released in September 2016. Common sources from the first data release have been combined with the Tycho-2 catalogue to provide a 5 parameter astrometric solution for approximately 2 million stars. The second Gaia data release is scheduled to come out in April 2018 and is expected to provide astrometry and photometry for more than 1 billion stars, a subset of which with a the full 6 parameter astrometric solution (adding radial velocity) and positional accuracy better than 0.002 arcsec (2 mas). In addition to precise astrometry, a unique opportunity exists with the Gaia catalogue in its production of accurate, broadband photometry using the Gaia G filter. In the past, clear filters have been used by various groups to maximize likelihood of detection of dim man-made objects but these data were very difficult to calibrate. With the second release of the Gaia catalogue, a ground based system utilizing the G band filter will have access to 1.5 billion all-sky calibration sources down to an accuracy of 0.02 magnitudes or better. In this talk, we will discuss the advantages and practicalities of implementing the Gaia filters and catalogue into data pipelines designed for optical observations of man-made objects.

Frith, James M.↗
DOE OSTI2020

The updated DESGW processing pipeline for the third LIGO/VIRGO observing run

The DESGW group seeks to identify electromagnetic counterpartsof gravitational wave events seen by the LIGO-VIRGO network, such as thoseexpected from binary neutron star mergers or neutron star- black hole mergers.DESGW was active throughout the first two LIGO observing seasons, followingup several binary black hole mergers and the first binary neutron star merger,GW170817. We describe the modifications to the observing strategy generationand image processing pipeline between the second (ending in August 2017)and third (beginning in April 2019) LIGO observing seasons. The modifica-tions include a more robust observing strategy generator, further parallelizationof the image reduction software and dierence imaging processing pipeline,data transfer streamlining, and a web page listing identified counterpart candi-dates that updates in real time. Taken together, the additional parallelizationsteps enable us to identify potential electromagnetic counterparts within fullycalibrated search images in less than one hour, compared to the 3-5 hours itwould typically take during the first two seasons. These performance improve-ments are critical to the entire EM followup community, as rapid identification(or rejection) of candidates enables detailed spectroscopic followup by multipleinstruments as soon as possible, leading to more information about the environ-ment immediately following such gravitational wave events.

Herner, Kenneth R.↗
DOE OSTI2020

Completion design improvement using a deep convolutional network

Maximizing stimulated natural and hydraulic fracture network is one of the primary hydraulic fracturing concerns for economic production from a horizontal shale gas well. Geomechanical facies and preexisting fractures in each stage are identified based on similarities in formation characteristics to optimize the locations of perforation clusters. This often requires analyzing large volumes of drilling, Logging While Drilling (LWD) and Measurement While Drilling (MWD) data. In this paper, we develop a methodology that calculates the mechanical specific energy (MSE) using real-time drill string acceleration signals directly from its definition. High resolution vibration signals have been collected using a tri-axial accerlometer, which was an auxiliary tool included in acoustic borehole imager. This technique provides a cost-efficient solution for engineered completion design. Furthermore, we adopt deep Convolutional Neural Network (CNN) with signal processing to build a data pipeline that effectively extracts patterns from dynamic acceleration signals for rock lateral MSE classification. First, we apply discrete wavelet transform and Short-Time Fourier Transform (STFT) for signal denoising and pattern recognition. Then we construct an image dataset using multi-scale image fusion at pixel level from 3 sensor channels, including axial, lateral acceleration spectrograms and zero-padded revolutions per minute (RPM). The resulted RGB image dataset includes 4,000 images of 5 MSE ranges with various rock strength conditions. Our results demonstrate that the proposed deep learning model can achieve more than 90% classification accuracy. The deep learning results, as a reference source, were applied in selected Marcellus Shale Energy and Environmental Lab (MSEEL) wells engineered completion located in the Marcellus shale gas site.

03 NATURAL GAS↗
DOE OSTI2026

HARMONY: Large-Scale Architecture Search for Efficient Hybrid Language Models

As large language models scale to trillions of parameters, their computational and memory requirements present critical challenges for efficient training and deployment. While Mixture of Experts (MoE) architectures enable efficient scaling through sparse parameter activation, and state-space models like Mamba offer linear-time complexity, principled methods for combining these paradigms remain undeveloped. We introduce HARMONY (Hybrid Architecture Research for Mamba, Optimized with Neural efficiencY), a multi-objective evolutionary neural architecture search framework for discovering efficient hybrid language models that integrate Transformer attention mechanisms, Mixture-of-Experts routing, and Mamba state-space components. Through large-scale distributed search using 16,384 MI250X GPUs on the Frontier supercomputer, HARMONY explores a comprehensive design space encompassing six attention variants (MHA, MQA, GQA, MLA, SWA, and Mamba-2), variable MoE configurations with both routed and shared experts, and extensive Mamba hyperparameters. Our framework discovers heterogeneous architectures that balance training performance with computational efficiency through multi-objective optimization incorporating latency penalties and fitness-based selection. Analysis of discovered architectures reveals that optimal hybrid designs favor heterogeneous component mixing rather than homogeneous patterns, with Mamba-2 and Multi-Head Latent Attention (MLA) emerging as preferred mechanisms. Discovered architectures demonstrate superior training efficiency: our best configuration achieves a final perplexity of 1.0874 with 2.38B parameters while processing 4,320 tokens/second, outperforming significantly larger manually designed models. Full-scale evaluation shows HARMONY's top architectures achieve better loss trajectories than equivalently-sized models using state-of-the-art configurations including Mixtral, Jamba, and Samba. Additionally, we demonstrate 91% weak scaling efficiency when training discovered 36B-parameter models across 1,024 GPUs. HARMONY is released as an open framework with comprehensive tools for building and training hybrid models using expert-data-pipeline parallelism, democratizing access to automated architecture design for next-generation language models.

Herron, Emily [ORNL] (ORCID:0000000273008172)↗
DOE OSTI2025

Summary of Carbon Dioxide Pipeline Systems and Incident Data in North America

Pipelines are historically seen as the primary transportation mode for carbon dioxide (CO 2 ) streams in the context of carbon capture and storage (CCS) and oil and gas industries. Pipeline transmission of CO 2 over longer distances is regarded as most efficient and economical when the CO 2 is in the dense phase, i.e., in liquid or supercritical regime, due to transporting CO 2 in dense phase that allows for a smaller-diameter pipeline to move a given flow, which optimizes project cost.

42 ENGINEERING↗
DOE OSTI2024

Hydropower Capacity Factor Trends & Analytics for the United States

This data repository contains all code, input data, and data generated for Turner et al. (2024)—“Hydropower capacity factors trending down in the United States”. File descriptions: – hydro-cf-trends-inputs.zip: Full set of input data used in this study, organized for direct entry into “/data” directory of hydro-cf-trends data processing pipeline. – hydro-cf-trends.zip: Full data processing pipeline, coded using the R {targets} framework. This is a snapshot release (v1.0) of the code repository stored at https://code.ornl.gov/turnersw/hydro-cf-trends/. – hydro-cf-trends-results.zip: Provides all dam level results required to reproduce results and graphics in Turner et al. (2024). Dams are identified by the “complxID” (root of the hydropower plant ID in the Existing Hydropower Assets Database, inherited from HILARRI). Results include: • dam_CF_trends.csv: Table of long-term trends in annualized capacity factors for 610 dams and modeled annualized capacity factors for 362 modeled dams (naturalized and assimilated flows). • dam_annualized_CF_gen.csv: Annualized time series of the following variables for each of 610 hydropower dams with nameplate > 5MW – Reported nameplate capacity (MW) – Implied maximum annual generation (MWh) – Reported net generation (MWh) – Computed annual capacity factor – Modeled annual capacity factor (362 modeled plants only)

13 HYDRO ENERGY↗
NASA NTRS2001

Using XML and Java Technologies for Astronomical Instrument Control

Traditionally, instrument command and control systems have been highly specialized, consisting mostly of custom code that is difficult to develop, maintain, and extend. Such solutions are initially very costly and are inflexible to subsequent engineering change requests, increasing software maintenance costs. Instrument description is too tightly coupled with details of implementation. NASA Goddard Space Flight Center, under the Instrument Remote Control (IRC) project, is developing a general and highly extensible framework that applies to any kind of instrument that can be controlled by a computer. The software architecture combines the platform independent processing capabilities of Java with the power of the Extensible Markup Language (XML), a human readable and machine understandable way to describe structured data. A key aspect of the object-oriented architecture is that the software is driven by an instrument description, written using the Instrument Markup Language (IML), a dialect of XML. IML is used to describe the command sets and command formats of the instrument, communication mechanisms, format of the data coming from the instrument, and characteristics of the graphical user interface to control and monitor the instrument. The IRC framework allows the users to define a data analysis pipeline which converts data coming out of the instrument. The data can be used in visualizations in order for the user to assess the data in real-time, if necessary. The data analysis pipeline algorithms can be supplied by the user in a variety of forms or programming languages. Although the current integration effort is targeted for the High-resolution Airborne Wideband Camera (HAWC) and the Submillimeter and Far Infrared Experiment (SAFIRE), first-light instruments of the Stratospheric Observatory for Infrared Astronomy (SOFIA), the framework is designed to be generic and extensible so that it can be applied to any instrument. Plans are underway to test the framework with other types of instruments, such as remote sensing earth science instruments.

Ames, Troy↗