SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

All sources NASA NTRS DOE OSTI

NASA NTRS2011

Software Displays Data on Active Regions of the Sun

The Solar Active Region Display System is a computer program that generates, in near real time, a graphical display of parameters indicative of the spatial and temporal variations of activity on the Sun. These parameters include histories and distributions of solar flares, active region growth, coronal mass ejections, size, and magnetic configuration. By presenting solar-activity data in graphical form, this program accelerates, facilitates, and partly automates what had previously been a time-consuming mental process of interpretation of solar-activity data presented in tabular and textual formats. Intended for original use in predicting space weather in order to minimize the exposure of astronauts to ionizing radiation, the program might also be useful on Earth for predicting solar-wind-induced ionospheric effects, electric currents, and potentials that could affect radio-communication systems, navigation systems, pipelines, and long electric-power lines. Raw data for the display are obtained automatically from the Space Environment Center (SEC) of the National Oceanic and Atmospheric Administration (NOAA). Other data must be obtained from the NOAA SEC by verbal communication and entered manually. The Solar Active Region Display System automatically accounts for the latitude dependence of the rate of rotation of the Sun, by use of a mathematical model that is corrected with NOAA SEC active-region position data once every 24 hours. The display includes the date, time, and an image of the Sun in H light overlaid with latitude and longitude coordinate lines, dots that mark locations of active regions identified by NOAA, identifying numbers assigned by NOAA to such regions, and solar-region visual summary (SRVS) indicators associated with some of the active regions. Each SRVS indicator is a small pie chart containing five equal sectors, each of which is color-coded to provide a semiquantitative indication of the degree of hazard posed by one aspect of the activity at the indicated location. The five aspects in question are the history of solar flares, the history of coronal mass ejections, the growth or decay of activity, the overall size, and the magnetic configuration. Mouse-clicking on an active-region-marking dot, SRVS indicator, or NOAA region number causes the program to generate a solar-region summary table (SRT) for the active region in question. The SRT contains additional quantitative and qualitative data, beyond those contained in the SRVS: These data include the solar coordinates of the region, the area of the region and its change in area during the past 24 hours, the change in the number of sunspots in the region during the past 24 hours, the magnetic configuration, and the types, dates, and times of the most recent flare and coronal mass ejection.

Golightly, Mike↗

NASA NTRS1997

The ASC Data Archive for the AXAF Ground Calibration

A data archive is near completion at the AXAF Science Center (ASC) to store and provide access to AXAF data. The archive is a distributed client/server system. It consists of a number of different servers which handle flat data files, relational data, replication across multiple sites, and the interface to the WWW. There is a 4GL client interface for each type of data server, C++ and Java APIs, and a number of standard clients to archive and retrieve data. The architecture is scalable and configurable in order to accommodate future data types and increasing data volumes. The first release of the system became available in August 1996 and has been successfully operated since then in support of the AXAF calibration at the George C. Marshall Space Flight Center, Huntsville, AL. This paper will present the overall archive architecture and the design of client and server components. It will also review the ground calibration data contents, the plan for flight data, and the access to the data by ASC internal users and processing pipelines.

Zografou, Panagoula↗

NASA NTRS2018

GEONEX: Challenges in Producing MODIS-Like Land Products from a New Generation of Geostationary Sensors

The new generation geostationary (GEO) remote sensors (GOES-R ABI, Himawari AHI, and FY4 AGRI) provide high frequency (5-15 minute) observations spatially/spectrally similar to MODIS/VIIRS for land monitoring. These new features of GEO satellite sensors make producing MODIS like land products for terrestrial monitoring possible. The NASA Earth Exchange (NEX) team developed the GEONEX pipeline that is containerized, deployable on NASA Pleiades supercomputer as well as public cloud platforms (e.g. AWS). The processing pipeline is designed to take Himawari Standard Data (HSD) and GOES-16 L1b to generate surface reflectance (SR) and other high-level land remote sensing products. In order to produce low-Earth-orbiting (LEO) remote sensing compatible land products, inter-comparison between Himawari AHI and MODIS Terra/Aqua has been conducted in this research work. Comparisons of TOA reflectance and surface reflectance between AHI and Terra/Aqua are presented. Ray-Matching method was used to locate the co-located pixels, where GEO and LEO sensors look at the land target with similar Viewing Zenith Angle (VZA) and Viewing Azimuth Angle (VAA) simultaneously. Here, we address challenges associated with the selection of qualified pixels of similar solar illumination condition and atmosphere path. We used strict criterion to constrain the pixel selection: the time difference between GEO and LEO observations is less than +-2.5 mins, the cosine of VZA difference is less than 1%, and the VAA difference is less than 10 deg. We also discuss the strong radiometric consistency that the new generation GEO sensors along with the popular LEO sensors would benefit the environmental remote sensing community.

Li, Shuang↗

NASA NTRS1982

Control structures for high speed processors

A special processor was designed to function as a Reed Solomon decoder with throughput data rate in the Mhz range. This data rate is significantly greater than is possible with conventional digital architectures. To achieve this rate, the processor design includes sequential, pipelined, distributed, and parallel processing. The processor was designed using a high level language register transfer language. The RTL can be used to describe how the different processes are implemented by the hardware. One problem of special interest was the development of dependent processes which are analogous to software subroutines. For greater flexibility, the RTL control structure was implemented in ROM. The special purpose hardware required approximately 1000 SSI and MSI components. The data rate throughput is 2.5 megabits/second. This data rate is achieved through the use of pipelined and distributed processing. This data rate can be compared with 800 kilobits/second in a recently proposed very large scale integration design of a Reed Solomon encoder.

Maki, G. K.↗

NASA NTRSDate not supplied

Science Target Prioritization Framework for Remote Sensing

Behind the scenes of a remote sensing mission there are complex decision making and planning operations. Streamlining these operations, with a quantitative scientific value framework, aids efficient and optimized science data collection.While there have been previous efforts to quantify the science value for specific science scenarios, our work aims to develop a general framework which can be applied across different scenarios. We describe a pipeline of processes which combines model forecast and observation data, in computational forms, as dictated by the mission objectives set forth by subject matter experts. The framework is described with use cases involving the monitoring of nitrogen dioxide (NO2) concentrations over the Gulf of Mexico and methane concentrations over interior Alaska.

Remote Sensing↗

NASA NTRS2010

Kepler Science Operations Center Pipeline Framework

The Kepler mission is designed to continuously monitor up to 170,000 stars at a 30 minute cadence for 3.5 years searching for Earth-size planets. The data are processed at the Science Operations Center (SOC) at NASA Ames Research Center. Because of the large volume of data and the memory and CPU-intensive nature of the analysis, significant computing hardware is required. We have developed generic pipeline framework software that is used to distribute and synchronize the processing across a cluster of CPUs and to manage the resulting products. The framework is written in Java and is therefore platform-independent, and scales from a single, standalone workstation (for development and research on small data sets) to a full cluster of homogeneous or heterogeneous hardware with minimal configuration changes. A plug-in architecture provides customized control of the unit of work without the need to modify the framework itself. Distributed transaction services provide for atomic storage of pipeline products for a unit of work across a relational database and the custom Kepler DB. Generic parameter management and data accountability services are provided to record the parameter values, software versions, and other meta-data used for each pipeline execution. A graphical console allows for the configuration, execution, and monitoring of pipelines. An alert and metrics subsystem is used to monitor the health and performance of the pipeline. The framework was developed for the Kepler project based on Kepler requirements, but the framework itself is generic and could be used for a variety of applications where these features are needed.

Klaus, Todd C.↗

NASA NTRSDate not supplied

Phenomena Portal: Large- Scale Visual Exploration of Atmospheric Phenomena

The Earth science community is experiencing a high influx of remote sensing data due to recent advancements in sensor technology. This enables the community to extend their research on a larger scale than ever before. Unfortunately, traditional data processing techniques do not scale well to these new, high volume data sources. State-of-the-art machine learning (ML) pipelines have been proven to overcome these burdens in various other fields but are underexploited within the physical sciences community. Moreover, ML is reliant on labeled data, which is currently sparsely available, owing to the fact that ML adoption is still in the early stages within the Earth and atmospheric science communities. To address these issues, we developed the Phenomena Portal, a visual exploration tool that uses ML to detect various atmospheric phenomena on a global scale. This allows the Earth and atmospheric science communities to view trends of occurrences of phenomena, identify potential relationships between them, and analyze spatiotemporal patterns over time. These detections can also serve as initial labeled data for ML research pertaining to the respective phenomena. The tool also incorporates feedback from subject matter experts to further improve the model detection accuracy, thereby facilitating human-in-the-loop. This presentation will provide an overview of the ML model development and cloud deployment. We also discuss the capabilities of the user interface for displaying the detections.

Muthukumaran Ramasubramanian↗

NASA NTRS2005

Video Mosaicking for Inspection of Gas Pipelines

A vision system that includes a specially designed video camera and an image-data-processing computer is under development as a prototype of robotic systems for visual inspection of the interior surfaces of pipes and especially of gas pipelines. The system is capable of providing both forward views and mosaicked radial views that can be displayed in real time or after inspection. To avoid the complexities associated with moving parts and to provide simultaneous forward and radial views, the video camera is equipped with a wide-angle (>165 ) fish-eye lens aimed along the axis of a pipe to be inspected. Nine white-light-emitting diodes (LEDs) placed just outside the field of view of the lens (see Figure 1) provide ample diffuse illumination for a high-contrast image of the interior pipe wall. The video camera contains a 2/3-in. (1.7-cm) charge-coupled-device (CCD) photodetector array and functions according to the National Television Standards Committee (NTSC) standard. The video output of the camera is sent to an off-the-shelf video capture board (frame grabber) by use of a peripheral component interconnect (PCI) interface in the computer, which is of the 400-MHz, Pentium II (or equivalent) class. Prior video-mosaicking techniques are applicable to narrow-field-of-view (low-distortion) images of evenly illuminated, relatively flat surfaces viewed along approximately perpendicular lines by cameras that do not rotate and that move approximately parallel to the viewed surfaces. One such technique for real-time creation of mosaic images of the ocean floor involves the use of visual correspondences based on area correlation, during both the acquisition of separate images of adjacent areas and the consolidation (equivalently, integration) of the separate images into a mosaic image, in order to insure that there are no gaps in the mosaic image. The data-processing technique used for mosaicking in the present system also involves area correlation, but with several notable differences: Because the wide-angle lens introduces considerable distortion, the image data must be processed to effectively unwarp the images (see Figure 2). The computer executes special software that includes an unwarping algorithm that takes explicit account of the cylindrical pipe geometry. To reduce the processing time needed for unwarping, parameters of the geometric mapping between the circular view of a fisheye lens and pipe wall are determined in advance from calibration images and compiled into an electronic lookup table. The software incorporates the assumption that the optical axis of the camera is parallel (rather than perpendicular) to the direction of motion of the camera. The software also compensates for the decrease in illumination with distance from the ring of LEDs.

Magruder, Darby↗

NASA NTRS2014

First All-Sky Search for Continuous Gravitational Waves from Unknown Sources in Binary Systems

We present the first results of an all-sky search for continuous gravitational waves from unknown spinning neutron stars in binary systems using LIGO and Virgo data. Using a specially developed analysis program, the TwoSpect algorithm, the search was carried out on data from the sixth LIGO science run and the second and third Virgo science runs. The search covers a range of frequencies from 20 Hz to 520 Hz, a range of orbital periods from 2 to ∼2,254 h and a frequency- and period-dependent range of frequency modulation depths from 0.277 to 100 mHz. This corresponds to a range of projected semimajor axes of the orbit from ∼0.6 × 10(exp −3) ls to ∼6,500 ls assuming the orbit of the binary is circular. While no plausible candidate gravitational wave events survive the pipeline, upper limits are set on the analyzed data. The most sensitive 95% confidence upper limit obtained on gravitational wave strain is 2.3 × 10(exp −24) at 217 Hz, assuming the source waves are circularly polarized. Although this search has been optimized for circular binary orbits, the upper limits obtained remain valid for orbital eccentricities as large as 0.9. In addition, upper limits are placed on continuous gravitational wave emission from the low-mass x-ray binary Scorpius X-1 between 20 Hz and 57.25 Hz.

systems↗

NASA NTRS2023

Uncovering Unique Molecular Adaptations in the Arabidopsis Thaliana Cvi-0 Ecotype

This research proposal aims to investigate the unique molecular adaptations exhibited by Arabidopsis Thaliana, specifically focusing on the Cape Verde Islands (Cvi-0) ecotype, in response to microgravity conditions. The study examines data from NASA’s Open Science Data Repository and applies a multifaceted RNAseq analysis pipeline using tools in the UseGalaxy.org open platform. Through transcriptomic analysis, differential gene expression patterns were identified in Cvi-0, revealing an absence of heat shock protein (HSP) upregulation and an upregulation of Rubisco Activase (RCA) and chloroplast-related pathways. To test the hypothesis that these adaptations may contribute to Cvi-0’s increased adaptability in microgravity, a three-fold experimental design is proposed. Four experimental groups will be cultivated under simulated microgravity and ground control conditions, including Cvi-0, Col-0, and genetically modified Col-0 with silenced HSP genes, and genetically modified Col-0 with upregulated RCA gene. Growth parameters will be measured to assess plant resilience, and RNA sequencing will provide transcriptomic data for pathway analysis. Anticipated outcomes include improved markers of plant health (mass, growth, etc.) of Cvi-0 in simulated microgravity and enhanced resilience in genetically altered Col-0 variants, providing insights into potential mechanisms of adaptation. This research would bear significance for space agriculture, nutrition for extended space missions, and sustainable terrestrial crop enhancement. Moreover, the insights gained could reshape crop engineering on Earth, enhancing robustness to climate induced stresses and bolstering global food security. The proposal’s trajectory blends scientific curiosity with practical applicability, forging a path towards sustainable food production and improving human exploration beyond our planet.

GL4HS↗

NASA NTRS1989

Multinode reconfigurable pipeline computer

A multinode parallel-processing computer is made up of a plurality of innerconnected, large capacity nodes each including a reconfigurable pipeline of functional units such as Integer Arithmetic Logic Processors, Floating Point Arithmetic Processors, Special Purpose Processors, etc. The reconfigurable pipeline of each node is connected to a multiplane memory by a Memory-ALU switch NETwork (MASNET). The reconfigurable pipeline includes three (3) basic substructures formed from functional units which have been found to be sufficient to perform the bulk of all calculations. The MASNET controls the flow of signals from the memory planes to the reconfigurable pipeline and vice versa. the nodes are connectable together by an internode data router (hyperspace router) so as to form a hypercube configuration. The capability of the nodes to conditionally configure the pipeline at each tick of the clock, without requiring a pipeline flush, permits many powerful algorithms to be implemented directly.

Nosenchuck, Daniel M.↗

NASA NTRSDate not supplied

Earth Science Data Processing With Nextflow

Earth science data processing tasks present many challenges. These tasks often process large input datasets and require scores of CPU-hours to generate results. All but the simplest tasks will be decomposed into a series of computational or data manipulation steps, also known as a scientific workflow. In order to reduce the burden of orchestrating and running the dependent processing steps, a workflow execution engine is required. This poster describes the lessons learned by the CLARREO Pathfinder (CPF) team while developing multiple scientific workflows and utilizing the open-source Nextflow engine to execute them in a cloud computing environment. The Nextflow engine is designed with the following stated goals: first, the engine does not dictate how individual steps in the task are implemented (i.e. it is language and interface agnostic); second, the engine supports easy configuration and modularity at the workflow level so that others can easily execute our workflows to reproduce results; lastly, the engine eases development by transparently scaling execution from local to remote environments. Nextflow was developed for the bioinformatics domain but is a good fit for other scientific workflows where the overall task is well-described by a dataflow diagram. The CPF team has developed Nextflow pipelines (i.e. scientific workflows) to simulate CLARREO radiance, generate large look-up tables for inter-calibration algorithms, and generate L4 intercalibration data products. These pipelines consume from single-digits to hundreds of thousands of CPU-hours. In the development and evolution of these pipelines we have discovered many design patterns, pitfalls, and solutions to common problems. Our goal is to demonstrate important aspects of how to design, implement, run, and ultimately share Nextflow pipelines in the domain of Earth science.

Aron D Bartle↗

NASA NTRSDate not supplied

Developing a Cloud-Based ArcGIS Image Service for TROPOMI Level 2 Data: Preprocessing, Transformation, and Publication

TROPOMI, the Tropospheric Monitoring Instrument aboard the Sentinel-5 Precursor satellite, provides high spatiotemporal resolution atmospheric measurements. It is essential for monitoring air quality, greenhouse gases, and other trace gases. Integrating TROPOMI Level 2 data into an ArcGIS Image Service marks a significant advancement in the accessibility and utility of satellite-derived environmental information for GIS applications. This paper details the methodology for building an ArcGIS Image Service tailored to handle TROPOMI Level 2 data, with a focus on preprocessing, transformation, and cloud-based publication. The workflow utilizes Python and ArcPy for data reformatting, reprojection, and updating, ensuring efficient processing and cloud-based notifications. By establishing a robust pipeline, the ArcGIS Image Service provides real-time access to TROPOMI Level 2 data, enabling users to visualize, analyze, and interpret atmospheric phenomena effectively.

Level 2 Data, ArcGIS, Image Service, ArcPy, OGC↗

NASA NTRS2022

MLtool: Universal Supervised Machine Learning Tool to Model Tabulated Data

Machine Learning (ML) is a subfield of Artificial Intelligence that gives computers the ability to learn from past data without being explicitly programmed. The predictive capabilities of ML models have already been used to facilitate several scientific breakthroughs. However, the practical application of ML is often limited due to the gaps in technical knowledge of its users. The common issue faced by many scientific researchers is the inability to choose the appropriate ML pipelines that are needed to treat real-world data, which is often sparse and noisy. To solve this problem, we have developed an automated Machine Learning tool (MLtool) that includes a set of ML algorithms and approaches to aid scientific researchers. The current version of MLtool is implemented as an object-oriented Python code that is easily extensible. It includes 44 different regression algorithms used to model data. MLtool helps users select the best model for their data, based on the scoring metrics used. Besides regression algorithms, MLtool also includes a suite of pre- and post-processing techniques such as missing value imputation, categorical variable encoding, input feature normalization, uncertainty quantification, exploratory data analysis (EDA), etc. MLtool was tested on several publicly available multi-dimensional data sets and was found capable of making accurate predictions.

Machine learning↗

NASA NTRS2022

MLtool Python Code

Machine Learning↗

NASA NTRS1981

Technology and future ground processing systems

Land-observing satellites with multiple thematic mappers will produce data at rates of 100 to 300 Mbps. When coupled with a high daily scene production rate, these rates will require new approaches to ground processing. Consideration is given here to future downlink rates and data volumes, and requirements peculiar to the future user community are discussed. The advanced technologies required to attain an operational system in the years 1985-1990 are considered, together with advances foreseen in communications, mass storage, bulk memories, and data processing. Using advanced devices, a centralized data processing system capable of handling the 100 Mbps data rate is described. New approaches, among them a parallel pipelined calibration front-end, real-time browse image production, a high bandwidth optical disk archive, regional image broadcast and massively parallel product production, are considered. A distributed system capable of handling the 300 Mbps data rate is then described. Designs for a hub system and a regional processing center are presented.

Wood, B. J.↗

NASA NTRS1979

Landsat-D data acquisition and processing

The Landsat-D system, in which S-band and X-band data links will provide real-time direct transmission of sensor data, is described. System performance objectives include input data quantity of 200 MSS scenes/day and 100 TM scenes/day by the Domestic Communication Satellite, and processing capability, in a pipeline mode, of 2.6 x 10 to the 11th input bits of sensor data per 16-hour day. Data processing techniques, including automated cloudcover assessment, control point library build system, radiometric calibration, and geometric correction are discussed.

Smith, P. L., Jr.↗

NASA NTRS1998

White Dwarf Pulsars

Work on NAG5-3288 ("White Dwarf Pulsars") has been fully integrated with the identically titled project NAG5-4734. The final report below is the same, since the data analysis and interpretative work are integrated, as are the resulting (previous and in-pipeline) publications. The proposal was designed to study pulse and orbital modulations in candidate DQ Herculis stars. Data on 5 stars were obtained. The best results were obtained on YY Draconis, which exhibited a strongly pulsed hard X-ray flux, and even suggested a transition between one-pole and two-pole emission during the course of the observation. This result is being readied for inclusion in a comprehensive study of YY Draconis. A strong pulsation appeared to be present also in H0857-242, but with a period of - 50 minutes, confusion with the first harmonic of the satellite's orbital frequency is possible. So that result is uncertain and is "on ice". A negative result was obtained on 4UO608-49 (V347 Pup), suggesting either that the X-ray identification is incorrect, or that the source is very transient. Finally, data was obtained on V1432 Aql and WZ Sge, respectively the slowest and fastest of these stars. Combined with the ASCA data, the high-energy data demonstrates the latter to contain a white dwarf rotating with P = 27.87 s (Patterson et al. 1998, PASP, 110, 403). Optical photometry contemporaneous with the X-ray data was obtained of V1432 Aql, in order to study the variations in the eclipse waveform. As anticipated, the width and centroid of the eclipse appeared to vary with the 50-day "supercycle".

Patterson, Joseph↗