Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
NASA NTRS2010

Kepler Mission's Focal Plane Characterization Models Implementation

The Kepler Mission photometer is an unusually complex array of CCDs. A large number of time-varying instrumental and systemic effects must be modeled and removed from the Kepler pixel data to produce light curves of sufficiently high quality for the mission to be successful in its planet-finding objective. After the launch of the spacecraft, many of these effects are difficult to remeasure frequently, and various interpolations over a small number of sample measurements must be used to determine the correct value of a given effect at different points in time. A library of software modules, called Focal Plane Characterization (FC) Models, is the element of the Kepler Science Data Pipeline (hereafter "pipeline") that handles this. FC, or products generated by FC, are used by nearly every element of the SOC processing chain. FC includes Java components: database persistence classes, operations classes, model classes, and data importers; and MATLAB code: model classes, interpolation methods, and wrapper functions. These classes, their interactions, and the database tables they represent, are discussed. This paper describes how these data and the FC software work together to provide the pipeline with the correct values to remove non-photometric effects caused by the photometer and its electronics from the Kepler light curves. The interpolation mathematics is reviewed, as well as the special case of the sky-to-pixel,pixel-to-sky coordinate transformation code, which incorporates a compound model that is unique in the SOC software.

mission↗
DOE OSTI2023

Digital Twin for Optimizing Real-time Economy of the Integrated Energy Systems

Economic and safe operation of integrated energy systems (IES) requires real-time optimization (RTO) of the control and actions conducted on each system component. In this regard, digital twins (DTs), which consist of a physical system, a virtual system, and the data communication that occurs between the two, are essential for effective RTO. Through the data warehouse, the virtual system is constantly updated with real-time data from the physical system, and functions as the model in the optimization framework. The reduced-order model of the dynamic process model in the virtual system is used in the optimization framework. The optimization results are then returned, via the data warehouse, as control actions to the physical system. This work demonstrates the software capabilities of DT assets for an IES in the context of preparing a DT for an experimental system comprised of Idaho National Laboratory (INL)’s Thermal Energy Delivery System and battery system. For the virtual demonstration, the DTs encompass (1) a physical system, including the Modelica models of the Thermal Energy Delivery System and the battery system; (2) virtual optimization via the Optimization of Real-Time Capacity Allocation (ORCA) platform; and (3) the open-source data warehouse software DeepLynx. This work assesses the performance of ORCA, which utilizes a reduced-order model built using the Risk Analysis Virtual Environment (RAVEN) and trained on the Modelica models and real-time data pipeline through the graph database hosted in DeepLynx. The proposed optimization workflow will be an RTO model based on DTs and the data they generate.

25 ENERGY STORAGE↗
DOE OSTI2021

Expanding Benchmarks for Nuclear Data Validation [Slides]

The presentation discusses benchmarking in comparison to experimental truth, the integral experiments that were performed, and that benchmarks are evaluated experiments. It also discusses validation testing, and that the ultimate goal of the validation end product is to improve evaluated nuclear data for applications It does state that validation highlights errors in the Nuclear Data Pipeline that affect applications, for example: Missing Cd Capture Gammas in ENDF/B-VII.1. It acknowledges that additional types of experiments are needed to test data used in applications, and that validation experiments do not have to be complicated and expensive. The presentation also discusses different types of measurements including activation foil and fission chamber measurements, reactor kinetics measurements, and subcritical measurements. The presentation concludes with an experiment summary and states that all applications need validation.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
NASA NTRS2018

The Zwicky Transient Facility: System Overview, Performance, and First Results

The Zwicky Transient Facility (ZTF) is a new optical time-domain survey that uses the Palomar 48 inch Schmidt telescope. A custom-built wide-field camera provides a 47 deg ^(2) field of view and 8 s readout time, yielding more than an order of magnitude improvement in survey speed relative to its predecessor survey, the Palomar Transient Factory. We describe the design and implementation of the camera and observing system. The ZTF data system at the Infrared Processing and Analysis Center provides near-real-time reduction to identify moving and varying objects. We outline the analysis pipelines, data products, and associated archive. Finally, we present on-sky performance analysis and first scientific results from commissioning and the early survey. ZTF’s public alert stream will serve as a useful precursor for that of the Large Synoptic Survey Telescope.

Eric C. Bellm↗
NASA NTRSDate not supplied

Ramdb: The NASA Raman Spectral Database (version 1.00).

Given that, in most instances, minimal sample preparation is required and due to its contactless instrument design, Raman spectroscopy is one of the most versatile vibrational spectroscopic techniques for the chemical analysis of environmental and biological specimens. The diversity of applications of Raman spectroscopy ranges anywhere from art [1] to planetary science missions [2]. The advancement in the use of Raman spectroscopy in Solar System missions, notably in post-mission sample return analysis, requires a spectral library holding the broad range of specimens that could be found in Solar System sources. For this purpose, we have initiated the development of a Raman spectral database (Ramdb) at NASA Ames Research Center. Currently, the database includes experimental and theoretical Raman spectra of PAHs [3, 4], as well as laboratory Raman spectra of amino acids, carbon allotropes, minerals, and analogs relevance to Earth Sciences [5], Exobiology [6], Planetary [7], and Astrochemistry [8] to name just a few examples. Ramdb can be found on the web at www.astrochemistry.org/ramdb, where raw and processed Raman spectra can be downloaded in CSV format. The laboratory Raman spectra are measured using a laser Raman spectrometer (JASCO NRS-5500-532QRI). The Raman instrument is equipped with three excitation lasers, with wavelengths of 405, 532, and 785 nm. A clean silicon substrate is used as the internal standard for wavenumber calibration. Powdered samples were prepared (microscopic >10 um, grounded microscopic < 10 um) on glass slides. Some raw data exhibited a background signal arising as a combination of laser-induced fluorescence from the sample. To correct this background, we developed a Python pipeline that uses open-source Python libraries. Ramdb provides both raw and processed (using Python pipeline) data, which includes tabulated Raman shift transitions and other measurement details. The theoretical Raman band positions of PAHs (pyrene monomers and tetramer clusters) were computed using density functional theory (DFT) with the help of the Gaussian 16 suite of programs [9]. In the near future, Ramdb will serve as a repository of Raman spectral data from Laboratory Astrophysics and Planetary Science experiments involving the irradiation of organic compounds under simulated space and planetary conditions. In addition, online and offline tools will be developed for utilising the database for comparison to the user’s sample.

N Punnakayathil↗
DOE OSTI2024

Navigating Exascale Operational Data Analytics: From Inundation to Insight

In this paper, we address the challenges in achieving sustainable data-driven efficiency by providing a detailed exploration of the end-to-end operational data analytics (ODA) framework that evolved through two generations of supercomputer systems at the Oak Ridge Leadership Computing Facility (OLCF). This framework addresses large data streams ingested from heavily instrumented HPC environment that accumulates multi-terabytes per day. We outline the multifaceted data life cycle across HPC procurement, operations, and research & development, identifying key obstacles and design decisions that shape effective strategies in building and supporting data pipelines end-to-end. By sharing key insights and lessons learned from our experience, we offer recommendations for the HPC community on enabling sustainable operational data analytics and beyond. Our contributions aim to bridge the gap between potential and real benefits of operational data, guiding future efforts towards integrated and sustainable operational intelligence in high-performance computing environments.

Shin, Woong↗
DOE OSTI2023

Report on ISR-1 High-Altitude Balloon Flight

To test small technologies at lower cost for space science applications, LANL has developed a small high altitude balloon payload that could, in the future, be regularly and inexpensively launched from LANL. A neutron detector, NEMO, was integrated to evaluate its performance in a space-like mixed-radiation environment and collect neutron data in the atmosphere. In collaboration with EES-14, a high-altitude balloon payload was launched from LANL Technical Area 51 on February 27, 2023 and April 17, 2023. For real-time geolocation, a SAM-M8Q M8 GNSS module was used to get position and time, and an Iridium RockBLOCK 9603 was used to communicate with the ground using the Iridium satellite fleet. These modules were all controlled using an Iteaduino Mega microcontroller board. Finally, a High Altitude Science Eagle Flight Computer with a temperature pressure sensor ran independently, writing data to an SD card. All of these modules were powered by a 5 mAh lithium polymer battery. The battery was attached to the bottom of the payload while the remaining electronics were embedded in the underside of the top of the payload. These modules were wired as seen in Figure 1-2. The Iteaduino Mega microcontroller board was programmed to use the RockBLOCK to send a message once every 10 minutes containing neutron and GPS data read off the NEMO and SAM M8Q, respectively. Once the message send attempt finished, the RockBLOCK would be slept for the rest of the 10 minute interval. The Eagle Flight Computer ran continuously throughout the flight, taking data every 6 seconds. The RockBLOCK message data was set up to be delivered from the Iridium satellite fleet to a website, where it was stored and parsed to create live maps and plots for analysis and balloon retrieval. The RockBLOCK message data was additionally configured to be sent to an email as a fail-safe. The payload was ground-tested successfully for over 50 hours, with multiple revisions occurring to best prepare for conditions at altitude and improve the software and firmware to fix any issues that cropped up with the data pipeline. Additional to the balloon payload, the flight had an attached iMet-4 radiosonde and Garmin T5 GPS Dog Collar. The radiosonde provided GPS and meteorological data. The T5 dog collar is used along with a Garmin Astro 430 to track the balloon at a range of up to 9 miles for retrieval. The balloon itself was initially a 1600 g meteorological balloon with an attached High Altitude Science parachute, both of which can be seen in Figure 1-3. After the first flight, the EES team swapped to a Rocketman parachute.

42 ENGINEERING↗
NASA NTRS1993

Avoiding and tolerating latency in large-scale next-generation shared-memory multiprocessors

A scalable solution to the memory-latency problem is necessary to prevent the large latencies of synchronization and memory operations inherent in large-scale shared-memory multiprocessors from reducing high performance. We distinguish latency avoidance and latency tolerance. Latency is avoided when data is brought to nearby locales for future reference. Latency is tolerated when references are overlapped with other computation. Latency-avoiding locales include: processor registers, data caches used temporally, and nearby memory modules. Tolerating communication latency requires parallelism, allowing the overlap of communication and computation. Latency-tolerating techniques include: vector pipelining, data caches used spatially, prefetching in various forms, and multithreading in various forms. Relaxing the consistency model permits increased use of avoidance and tolerance techniques. Each model is a mapping from the program text to sets of partial orders on program operations; it is a convention about which temporal precedences among program operations are necessary. Information about temporal locality and parallelism constrains the use of avoidance and tolerance techniques. Suitable architectural primitives and compiler technology are required to exploit the increased freedom to reorder and overlap operations in relaxed models.

Probst, David K.↗
NASA NTRS1985

Real-Time Reed-Solomon Decoder

RS decoder uses dedicated hardware and data pipelining for high-speed operation. Parallel processing techniques provide equivalent of over one billion operations per second at one step in decoding. Decoder finds commercial application in data encoding/decoding, telemetry, and radio communications.

Lahmeyer, C. R.↗
NASA NTRSDate not supplied

Primary Mission Threshold Crossing Events in the TESS SPOC Transit Search

We present an overview of the single- and multiple-sector results of the Science Processing Operations Center (SPOC) transit search in the primary Transiting Exoplanet Survey Satellite (TESS) mission. TESS was designed to survey bright stars in the greater Solar neighborhood in search of transiting exoplanets. Data were acquired at a 2-minute cadence for 16,000-20,000 pre-selected target stars in each 28-day observation sector and processed in the SPOC pipeline at NASA Ames Research Center. The photometry pipeline produced a systematic error corrected light curve for each target star. Light curves were searched for transiting planet signatures by sector for all target stars, and separately for target stars observed in multiple sectors. Potential transit signals for which the transiting planet detection threshold was exceeded and a series of transit consistency tests were passed are referred to as Threshold Crossing Events (TCEs). We highlight the full TCE population and the population of SPOC TCEs that were later identified as TESS Objects of Interest (TOIs). Characteristics of the TCE populations implied by limb-darkened transiting planet model fits are also presented. SPOC pipeline data products are delivered to the Mikulski Archive for Space Telescopes (MAST)(http://archive.stsci.edu/missions-and-data/tess) for access by the community. Funding for the TESS Mission has been provided by the NASA Science Mission Directorate.

TESS↗
DOE OSTI2024

Online Detection of Power Grid Anomalies via Federated Learning

Data from sensors is critical for advanced applica- tions that support efficient, reliable, and resilient electric grid operations. Historically, data from phasor measurement units (PMU) has been utilized to develop a wide variety of wide area control and protection applications suitable for power grid control centers. However, until now, most of these could not be deployed for automated operations due to a set of data corruption challenges and uncertainty in the incoming data pipeline. In this paper, we address the problem of detecting different variety of anomalies that are evident in different high- speed power grid measurements. The paper discusses a workflow for handling problems with data acquisition and highlights some of the key findings suitable for anomaly detection in a centralized and distributed environment. The effectiveness of the proposed method was demonstrated with results utilizing realistic PMU datasets

Shinkle, Matthew W.↗
DOE OSTI2024

Automated pipeline processing X-ray diffraction data from dynamic compression experiments on the Extreme Conditions Beamline of PETRA III

Presented and discussed here is the implementation of a software solution that provides prompt X-ray diffraction data analysis during fast dynamic compression experiments conducted within the dynamic diamond anvil cell technique. It includes efficient data collection, streaming of data and metadata to a high-performance cluster (HPC), fast azimuthal data integration on the cluster, and tools for controlling the data processing steps and visualizing the data using the DIOPTAS software package. This data processing pipeline is invaluable for a great number of studies. The potential of the pipeline is illustrated with two examples of data collected on ammonia–water mixtures and multiphase mineral assemblies under high pressure. The pipeline is designed to be generic in nature and could be readily adapted to provide rapid feedback for many other X-ray diffraction techniques, e.g. large-volume press studies, in situ stress/strain studies, phase transformation studies, chemical reactions studied with high-resolution diffraction etc.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1992

A statistical-based scheduling algorithm in automated data path synthesis

In this paper, we propose a new heuristic scheduling algorithm based on the statistical analysis of the cumulative frequency distribution of operations among control steps. It has a tendency of escaping from local minima and therefore reaching a globally optimal solution. The presented algorithm considers the real world constraints such as chained operations, multicycle operations, and pipelined data paths. The result of the experiment shows that it gives optimal solutions, even though it is greedy in nature.

Jeon, Byung Wook↗
NASA NTRS2001

Processors, Pipelines, and Protocols for Advanced Modeling Networks

Predictive capabilities arise from our understanding of natural processes and our ability to construct models that accurately reproduce these processes. Although our modeling state-of-the-art is primarily limited by existing computational capabilities, other technical areas will soon present obstacles to the development and deployment of future predictive capabilities. Advancement of our modeling capabilities will require not only faster processors, but new processing algorithms, high-speed data pipelines, and a common software engineering framework that allows networking of diverse models that represent the many components of Earth's climate and weather system. Development and integration of these new capabilities will pose serious challenges to the Information Systems (IS) technology community. Designers of future IS infrastructures must deal with issues that include performance, reliability, interoperability, portability of data and software, and ultimately, the full integration of various ES model systems into a unified ES modeling network.

Coughlan, Joseph↗
DOE OSTI2025

The DECADE cosmic shear project III: validation of analysis pipeline using spatially inhomogeneous data

We present the pipeline for the cosmic shear analysis of the Dark Energy Camera All Data Everywhere (DECADE) weak lensing dataset: a catalog consisting of 107 million galaxies observed by the Dark Energy Camera (DECam) in the northern Galactic cap. The catalog derives from a large number of disparate observing programs and is therefore more inhomogeneous across the sky compared to existing lensing surveys. First, we use simulated data-vectors to show the sensitivity of our constraints to different analysis choices in our inference pipeline, including sensitivity to residual systematics. Next we use simulations to validate our covariance modeling for inhomogeneous datasets. Finally, we show that our choices in the end-to-end cosmic shear pipeline are robust against inhomogeneities in the survey, by extracting relative shifts in the cosmology constraints across different subsets of the footprint/catalog and showing they are all consistent within 1σ to 2σ. This is done for forty-six subsets of the data and is carried out in a fully consistent manner: for each subset of the data, we re-derive the photometric redshift estimates, shear calibrations, survey transfer functions, the data vector, measurement covariance, and finally, the cosmological constraints. Our results show that existing analysis methods for weak lensing cosmology can be fairly resilient towards inhomogeneous datasets. This also motivates exploring a wider range of image data for pursuing such cosmological constraints.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2012

Using Quality Attributes to Bridge Systems Engineering Gaps : A Juno Ground Data Systems Case Study

The Juno Mission to Jupiter is the second mission selected by the NASA New Frontiers Program. Juno launched August 2011 and will reach Jupiter July 2016. Juno's payload system is composed of nine instruments plus a gravity science experiment. One of the primary functions of the Juno Ground Data System (GDS) is the assembly and distribution of the CFDP (CCSDS File Delivery Protocol) product telemetry, also referred to as raw science data, for eight out of the nine instruments. The GDS accomplishes this with the Instrument Data Pipeline (IDP). During payload integration, the first attempt to exercise the IDP in a flight like manner revealed that although the functional requirements were well understood, the system was unable to meet latency requirements with the as-is heritage design. A systems engineering gap emerged between Juno instrument data delivery requirements and the assumptions behind the heritage flight-ground interactions. This paper describes the use of quality attributes to measure and overcome this gap by introducing a new systems engineering activity, and a new monitoring service architecture that successfully delivered the performance metrics needed to validate Juno IDP.

Ground Data Systems (GDS)↗
DOE OSTI2022

Tsdat: An Open-Source Data Standardization Framework for Marine Energy and Beyond: Preprint

Many organizations are tasked with the collection and processing of large quantities of data from various measurement devices. Data reported from these sources are often not interoperable with datasets and software used by analysts and other organizations in the same field, introducing barriers for collaboration on large-scale projects. This poses a particular problem for cross-device comparisons and machine learning applications. To address these challenges, the open source Time-Series Data Pipelines (Tsdat) software was developed by a joint collaboration between Pacific Northwest National Laboratory, the National Renewable Energy Laboratory, and Sandia National Laboratories to facilitate collaboration and accelerate advancements in the Marine Energy domain through the development of an open-source ecosystem of tools. This paper will describe the Tsdat software and the data standards within which the framework operates. A beta version of the framework has been released and is currently being used by several projects in marine energy, wind energy, and building energy systems.

big data↗
DOE OSTI2023

Phonon-informed Neural Thermal Scattering (NeTS) Optimization for Crystalline Graphite and Beryllium Metal

Fast neutrons born from fission lose energy through scattering interactions in the process of slowing-down. As neutrons thermalize to the order of $k$ $b$ $T$ (where $k$ $b$ is the Boltzmann constant, and $T$ is the temperature of the medium), their de Broglie wavelength and energy approaches the order of inter-atomic spacing and quantized lattice vibrations, i.e., phonons. At thermal energies, the thermal scattering law (TSL), i.e., $S$($α, β$), captures crystal binding contributions to the total reaction rate, or cross section. This dimensionless material property describes the energy ($β$) and momentum ($α$) exchanges available in a medium. Currently, $S$($α, β$) is evaluated in the Full Law Analysis Scattering System Hub (FLASSH) code for discrete inputs and stored as ENDF/B File 7 for 0-phonon elastic (MT 2) and n-phonon inelastic (MT 4) processes. Further processing recasts $S$($α, β$) into cumulative distribution functions for sampling post-collision scattering kinematics. In practice, interpolation schemes are employed to access data between tabulated values. An improvement to this juncture of the nuclear data pipeline is supplying cross sections on-the-fly (OTF), as has been developed for the un-resolved resonance region to minimize non-physical interpolation errors. This capability may improve simulation accuracy for accident and transient analyses, where rapidly varying changes in temperature and pressure are difficult to predict beforehand. To do so, deep artificial neural networks (ANNs) can be employed which collapse non-linear, complex data into a lightweight dictionary of neural weights and biases. This has been successfully demonstrated for the hydrogen in light water $S$($α, β$) dataset in the form of a Neural Thermal Scattering (NeTS) module. In this work, the NeTS framework is extended to consider the impact of material-dependent dynamical features on optimal neural pre-processing and architecture design decisions, such as number of neurons per hidden layer, residual skip connections and neural depth. New NeTS modules for crystalline graphite and beryllium metal illuminate a novel correlation between dynamical nonlinearity and optimal neural parametrization when deploying $S$($α, β$) on-the-fly.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗