Search NASA⌕ Search

SEARCH · Search NASA

Results for “process data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,405 records · Page 78
All sourcesNASA NTRSDOE OSTI
NASA NTRS2022

Ozone Monitoring Instrument (OMI) collection 4: establishing a 17-year-long series of detrended level-1b data

The Ozone Monitoring Instrument (OMI) was launched on 15 July 2004, with an expected mission lifetime of 5 years. After more than 17 years in orbit the instrument is still functioning satisfactorily and in principle can continue doing so until the expected decommissioning of its platform Aura in 2025. In order to continue the datasets acquired by OMI and the Microwave Limb Sounder, the mission was extended up to at least 2023. Actions have been taken to ensure the proper functioning of the OMI operations, the data processing, and the calibration monitoring system until the eventual end of the mission. For the data processing a new level-0 (L0) to level-1b (L1b) data processor was built based on the recent developments for the TROPOspheric Monitoring Instrument (TROPOMI). With corrections for the degradation of the instrument now included, it is feasible to generate a new data collection to supersede the current collection-3 data products and reprocess the data of the entire mission up to now. This paper describes the differences between the collection-3 and collection-4 data. It will be shown that the collection-4 L1b data comprise a clear improvement with respect to the previous collections. By correcting for the gentle optical and electronic aging that has occurred over the past 17 years, OMI’s ability to make trend-quality ozone measurements has further improved.

UV↗
DOE OSTI2022

Hod Carrier

SAND2023-06720O Hod Carrier is a simple proof-of-concept library for moving data from host to NVIDIA BlueField device memory using remote direct memory access. The Hod Carrier software also facilitates research activities related to the potential uses of NVIDIA BlueField data processing unit devices. Using well-established technologies (e.g., RDMA over Infiniband), it will move data from the host to the data processing unit’s memory. This software is a middleware utility for moving data without regard to its semantic meaning. Its operation is fundamentally unaffected by storage allocation. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

SciDAC↗
NASA NTRS2003

Refinement of Protocols for Measuring the Apparent Optical Properties of Seawater

Ocean color satellite missions, like the Sea-viewing Wide Field-of-view Sensor (SeaWiFS) or the Moderate Resolution Imaging Spectroradiometer (MODIS) projects, are tasked with acquiring a global ocean color data set, validating and monitoring the accuracy and quality of the data, processing the radiometric data into geophysical units using a set of atmospheric and bio-optical algorithms, and distributing the final products to the scientific community. The long-standing requirement of the SeaWiFS Project, for example, is to produce spectral water-leaving radiances, LW(lambda), to within 5% absolute (lambda denotes wavelength) and chlorophyll a concentrations to within 35% (Hooker and Esaias 1993), and most ocean color sensors have the same or similar requirements. Although a diverse set of activities are required to ensure the accuracy requirements are met (Hooker and McClain 2000), the perspective here is with field observations. The accurate determination of upper ocean apparent optical properties (AOPs) is essential for the vicarious calibration of ocean color data and the validation of the derived data products, because the sea-truth measurements are used to evaluate the satellite observations (Hooker and McClain 2000). The uncertainties with in situ AOP measurements have various sources: a) the sampling procedures used in the field, including the environmental conditions encountered; b) the absolute characterization of the radiometers in the laboratory; c) the conversion of the light signals to geophysical units in a processing scheme, and d) the stability of the radiometers in the harsh environment they are subjected to during transport and use. Assuming ideal environmental conditions, so this aspect can be neglected, the SeaWiFS ground-truth uncertainty budget can only be satisfied if each uncertainty is on the order of 1-2%, or what is generally referred to as 1% radiometry. In recent years, progress has been made in estimating the magnitude of some of these uncertainties and in defining procedures for minimizing them. For the SeaWiFS Project, the first step was to convene a workshop to draft the SeaWiFS Ocean Optics Protocols (hereafter referred to as the Protocols). The Protocols initially adhered to the Joint Global Ocean Flux Study (JGOFS) sampling procedures (JGOFS 1991) and defined the standards for optical measurements to be used in SeaWiFS calibration and validation activities (Mueller and Austin 1992). Over time, the Protocols were revised (Mueller and Austin 1995), and then recurringly updated on essentially an annual basis (Mueller 2000, 2002, and 2003) as part of the Sensor Inter-comparison and Merger for Biological and Interdisciplinary Oceanic Studies (SIMBIOS) project. 98

Hooker, Stanford B.↗
NASA NTRS1984

Geocoded data structures and their applications to Earth science investigations

A geocoded data structure is a means for digitally representing a geographically referenced map or image. The characteristics of representative cellular, linked, and hybrid geocoded data structures are reviewed. The data processing requirements of Earth science projects at the Goddard Space Flight Center and the basic tools of geographic data processing are described. Specific ways that new geocoded data structures can be used to adapt these tools to scientists' needs are presented. These include: expanding analysis and modeling capabilities; simplifying the merging of data sets from diverse sources; and saving computer storage space.

Goldberg, M.↗
DOE OSTI2024

Grid Edge Waveform Analytics Framework for Event Detection and Classification

This paper provides a grid edge waveform analytics framework for power system event detection and classification in the local as well as in the wide area. This framework overviews data excellence for event detection and classification. The data excellence describes the data acquisition process and requirements, data processing, data quality, and data integrity. Power system event detection in the local area based on different features such as energy-based, cyclostationary approach, template matching, and wavelet transform are also discussed. Furthermore, local area event detection and classification using approaches such as statistical, signal processing, artificial intelligence, and hybrid are also discussed. Moreover, an overview of wide-area event detection and classification along with several other aspects such as wide-area events, wide-area event detection approaches, event location and system performance, event pattern recognition, inter-area oscillation, and wide-area frequency response under variable deployment of inverter-based resources are also provided. The proposed framework is the first step toward the goal of developing appropriate tools and methodologies to detect and classify local as well as wide-area events using waveform analytics. The appropriate event detection and classification framework development is especially important now as more and more grid edge devices with communication capabilities are being deployed in the modern power grid than ever before.

Bhusal, Narayan↗
NASA NTRS2001

Ocean Color Optical Property Data Derived from OCTS and POLDER: A Comparison Study

We describe our efforts in studying and comparing the ocean color data derived from the Japanese Ocean Color and Temperature Scanner (OCTS) and the French Polarization and Directionality of the Earth's Reflectances (POLDER). OCTS and POLDER were both on board Japan's Sun-synchronous Advanced Earth Observing Satellite (ADEOS-1) from August 1996 to June 1997, collecting about 10 months of global ocean color data. This provides a unique opportunity for developing methods and strategies for the merging of ocean color data from multiple ocean color sensors. In this paper, we describe our approach in developing consistent data processing algorithms for both OCTS and POLDER and using a common in situ data set to vicariously calibrate the two sensors. Therefore, the OCTS and POLDER-measured radiances are effectively bridged through common in situ measurements. With this approach in processing data from two different sensors, the only differences in the derived products from OCTS and POLDER are the differences inherited from the instrument characteristics. Results show that there are no obvious bias differences between the OCTS and POLDER-derived ocean color products, whereas the differences due to noise, which stem from variations in sensor characteristics, are difficult to correct. It is possible, however, to reduce noise differences with some data averaging schemes. The ocean color data from OCTS and POLDER can therefore be compared and merged in the sense that there is no significant bias between two.

Wang, Menghua↗
NASA NTRS2007

Robotic Vision-Based Localization in an Urban Environment

A system of electronic hardware and software, now undergoing development, automatically estimates the location of a robotic land vehicle in an urban environment using a somewhat imprecise map, which has been generated in advance from aerial imagery. This system does not utilize the Global Positioning System and does not include any odometry, inertial measurement units, or any other sensors except a stereoscopic pair of black-and-white digital video cameras mounted on the vehicle. Of course, the system also includes a computer running software that processes the video image data. The software consists mostly of three components corresponding to the three major image-data-processing functions: Visual Odometry This component automatically tracks point features in the imagery and computes the relative motion of the cameras between sequential image frames. This component incorporates a modified version of a visual-odometry algorithm originally published in 1989. The algorithm selects point features, performs multiresolution area-correlation computations to match the features in stereoscopic images, tracks the features through the sequence of images, and uses the tracking results to estimate the six-degree-of-freedom motion of the camera between consecutive stereoscopic pairs of images (see figure). Urban Feature Detection and Ranging Using the same data as those processed by the visual-odometry component, this component strives to determine the three-dimensional (3D) coordinates of vertical and horizontal lines that are likely to be parts of, or close to, the exterior surfaces of buildings. The basic sequence of processes performed by this component is the following: 1. An edge-detection algorithm is applied, yielding a set of linked lists of edge pixels, a horizontal-gradient image, and a vertical-gradient image. 2. Straight-line segments of edges are extracted from the linked lists generated in step 1. Any straight-line segments longer than an arbitrary threshold (e.g., 30 pixels) are assumed to belong to buildings or other artificial objects. 3. A gradient-filter algorithm is used to test straight-line segments longer than the threshold to determine whether they represent edges of natural or artificial objects. In somewhat oversimplified terms, the test is based on the assumption that the gradient of image intensity varies little along a segment that represents the edge of an artificial object.

Mchenry, Michael↗
NASA NTRS2009

Multi-Mission Automated Task Invocation Subsystem

Multi-Mission Automated Task Invocation Subsystem (MATIS) is software that establishes a distributed data-processing framework for automated generation of instrument data products from a spacecraft mission. Each mission may set up a set of MATIS servers for processing its data products. MATIS embodies lessons learned in experience with prior instrument- data-product-generation software. MATIS is an event-driven workflow manager that interprets project-specific, user-defined rules for managing processes. It executes programs in response to specific events under specific conditions according to the rules. Because requirements of different missions are too diverse to be satisfied by one program, MATIS accommodates plug-in programs. MATIS is flexible in that users can control such processing parameters as how many pipelines to run and on which computing machines to run them. MATIS has a fail-safe capability. At each step, MATIS captures and retains pertinent information needed to complete the step and start the next step. In the event of a restart, this information is retrieved so that processing can be resumed appropriately. At this writing, it is planned to develop a graphical user interface (GUI) for monitoring and controlling a product generation engine in MATIS. The GUI would enable users to schedule multiple processes and manage the data products produced in the processes. Although MATIS was initially designed for instrument data product generation,

Cheng, Cecilia S.↗
NASA NTRS1979

A rocket-borne pulse-height analyzer for energetic particle measurements

The pulse-height analyzer basically resembles a time-sharing multiplexing data-acquisition system which acquires analog data (from energetic particle spectrometers) and converts them into digital code. The PHA simultaneously acquires pulse-height information from the analog signals of the four input channels and sequentially multiplexes the digitized data to a microprocessor. The PHA together with the microprocessor form an on-board real-time data-manipulation system. The system processes data obtained during the rocket flight and reduces the amount of data to be sent back to the ground station. Consequently the data-reduction process for the rocket experiments is speeded up. By using a time-sharing technique, the throughput rate of the microprocessor is increased. Moreover, data from several particle spectrometers are manipulated to share one information channel; consequently, the TM capacity is increased.

Leung, W.↗
DOE OSTI2022

Machine learning on FPGA for event selection

Real-time data processing is a frontier field in experimental particle physics. The application of FPGAs at the trigger level is used by many current and planned experiments (CMS, LHCb, Belle2, PANDA). Usually they use conventional processing algorithms. LHCb has implemented Machine Learning (ML) elements for real-time data processing with a triggered readout system that runs most of the ML algorithms on a computer farm. The work described in this article aims to test the ML-FPGA algorithms for streaming data acquisition. Herein, there are many experiments working in this area and they have a lot in common, but there are many specific solutions for detector and accelerator parameters that are worth exploring further. This report describes the purpose of the work and progress in evaluating the ML-FPGA application.

47 OTHER INSTRUMENTATION↗
DOE OSTI2019

SKA shakes hands with Summit

Recently, a full-scale data processing workflow (Fig. 1) of the Square Kilometre Array (SKA) Phase 1 was successfully executed on the world’s fastest supercomputer Summit, proving that scientists have the expertise, software tools and computing resources to process the SKA data.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

Author Correction: US oil and gas system emissions from nearly one million aerial site measurements

Correction to: Naturehttps://doi.org/10.1038/s41586-024-07117-5 Published online 13 March 2024 In the version of the article initially published, several errors were present and have been corrected in the HTML and PDF versions of the article and Supplementary Information. The main results, conclusions, and our interpretations of the data remain unchanged. See the new Supplementary Information Section S15 for a more detailed description of the errors corrected and the resulting effects on the analysis. Data processing and methods corrections Overflight count correction: We previously used pre-computed source coverage data for some Carbon Mapper campaigns that was computed differently than was required for our analysis. We have re-computed Carbon Mapper source coverage based on flightline polygons and source coordinates. Transition point computation, well sites: The updated version now correctly compares the cumulative emissions distribution of simulated well site emissions with that of aerially detected sources (rather than plumes) when computing the transition point. Transition point computation, midstream: Additionally, the transition point calculation has been corrected to exclude aerially detected midstream emissions below the transition point, which was previously leading to double counting of these emissions. This error was not present for upstream (well site) emissions. Calculation errors Unit error: We corrected a specific unit conversion error affecting well site emissions in the Kairos Fort Worth dataset. Across all datasets, we also correct the conversion factor for converting from standard volume to mass for midstream emissions. Sorting error: We correct code that was applying incorrect sorting when computing correction factors to account for partial detection at well sites. Small typographical corrections were made in Fig. 1b and SI Section S4.1. Data processing and methods corrections Overflight count correction: We previously used pre-computed source coverage data for some Carbon Mapper campaigns that was computed differently than was required for our analysis. We have re-computed Carbon Mapper source coverage based on flightline polygons and source coordinates. Transition point computation, well sites: The updated version now correctly compares the cumulative emissions distribution of simulated well site emissions with that of aerially detected sources (rather than plumes) when computing the transition point. Transition point computation, midstream: Additionally, the transition point calculation has been corrected to exclude aerially detected midstream emissions below the transition point, which was previously leading to double counting of these emissions. This error was not present for upstream (well site) emissions. Calculation errors Unit error: We corrected a specific unit conversion error affecting well site emissions in the Kairos Fort Worth dataset. Across all datasets, we also correct the conversion factor for converting from standard volume to mass for midstream emissions. Sorting error: We correct code that was applying incorrect sorting when computing correction factors to account for partial detection at well sites. Small typographical corrections were made in Fig. 1b and SI Section S4.1. The following practices may help researchers conducting similar analyses avoid making similar errors: 1, Clear, accessible documentation explaining the interpretation of all columns in data input tables and all internal variables within the model, 2, Simple cross-check calculations computed before and after unit conversions.

Sherwin, Evan D↗
DOE OSTI2019

InP AAC for data compression applications

The authors propose a 2 × 2 asymmetric adiabatic coupler (AAC) indium phosphide (InP) photonic integrated circuit (PIC), suitable for tunable power applications such as all‐optical data processing. The device was idealised as the key building block in Haar transform network used for image compression, allowing to perform the necessary separate addition and subtraction of incoming input signals at its output ports. The tunable behaviour of the coupler was demonstrated experimentally for addition/subtraction and splitting (with experimental coupling ratios of 77:23/14:86 and 50:50, respectively) through phase control. The use of the developed InP PIC enables a low footprint structure design together with the high‐quality active components (lasers, photodetectors and phase modulators) of Fraunhofer Gesellschaft Heinrich Hertz Institute design toolkit. The proposed study provides a full experimental characterisation of the AAC, exploring its tuning properties and enabling further usage in a plethora of applications in high processing computing and data communications.

Pinho, Cátia↗
DOE OSTI2020

Planck 2018 results - IV. Diffuse component separation

We present full-sky maps of the cosmic microwave background (CMB) and polarized synchrotron and thermal dust emission, derived from the third set of Planck frequency maps. These products have significantly lower contamination from instrumental systematic effects than previous versions. The methodologies used to derive these maps follow closely those described in earlier papers, adopting four methods (Commander, NILC, SEVEM, and SMICA) to extract the CMB component, as well as three methods (Commander, GNILC, and SMICA) to extract astrophysical components. Our revised CMB temperature maps agree with corresponding products in the Planck 2015 delivery, whereas the polarization maps exhibit significantly lower large-scale power, reflecting the improved data processing described in companion papers; however, the noise properties of the resulting data products are complicated, and the best available end-to-end simulations exhibit relative biases with respect to the data at the few percent level. Using these maps, we are for the first time able to fit the spectral index of thermal dust independently over 3° regions. We derive a conservative estimate of the mean spectral index of polarized thermal dust emission of β d = 1.55 ± 0.05, where the uncertainty marginalizes both over all known systematic uncertainties and different estimation techniques. For polarized synchrotron emission, we find a mean spectral index of β s = -3.1 ± 0.1, consistent with previously reported measurements. We note that the current data processing does not allow for construction of unbiased single-bolometer maps, and this limits our ability to extract CO emission and correlated components. The foreground results for intensity derived in this paper therefore do not supersede corresponding Planck 2015 products. For polarization the new results supersede the corresponding 2015 products in all respects.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Euclid I. Overview of the Euclid mission

The current standard model of cosmology successfully describes a variety of measurements, but the nature of its main ingredients, dark matter and dark energy, remains unknown. Euclid is a medium-class mission in the Cosmic Vision 2015–2025 programme of the European Space Agency (ESA) that will provide high-resolution optical imaging, as well as near-infrared imaging and spectroscopy, over about 14 000 deg 2 of extragalactic sky. In addition to accurate weak lensing and clustering measurements that probe structure formation over half of the age of the Universe, its primary probes for cosmology, these exquisite data will enable a wide range of science. This paper provides a high-level overview of the mission, summarising the survey characteristics, the various data-processing steps, and data products. We also highlight the main science objectives and expected performance.

cosmology↗
DOE OSTI2020

DUNE DAQ R&D integration in ProtoDUNE Single-Phase at CERN

The DAQ system of ProtoDUNE-SP successfully proved its design principles and met the requirements of the beam run of 2018. The technical design of the DAQ system for the DUNE experiment has major differences compared to the prototype due to different requirements placed on the detector, as well as a radically different location of operation. The single-phase prototype at CERN is a major integration facility for R&D aspects of the DUNE DAQ system. The facility allows for the exploration of additional data processing capabilities and optimization of the FELIX system, which is the chosen TPC readout solution for the DUNE single-phase detectors. One of the fundamental differences from the prototype is that the DUNE DAQ relies on self-triggering. Therefore, real-time processing of the data stream for hit and trigger primitive finding is essential for the requirement of continuous readout. The supernova burst trigger requires a large and fast buffering technique, where 3D XPoint persistent memory solutions are evaluated and integrated. In order to maximize resource utilization of the FELIX hosting servers, the elimination of the 100 Gb network communication stack is desired. This implies the design and development of a single-host application layer, which is a fundamental element of the self-triggering chain. This paper discusses the evaluation and integration of these developments for the DUNE DAQ, in the ProtoDUNE environment.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

Towards an Introspective Dynamic Model of Globally Distributed Computing Infrastructures

Large-scale scientific collaborations like ATLAS, Belle II, CMS, DUNE, and others involve hundreds of research institutes and thousands of researchers spread across the globe. These experiments generate petabytes of data, with volumes soon expected to reach exabytes. Consequently, there is a growing need for computation, including structured data processing from raw data to consumer-ready derived data, extensive Monte Carlo simulation campaigns, and a wide range of end-user analysis. To manage these computational and storage demands, centralized workflow and data management systems are implemented. However, decisions regarding data placement and payload allocation are often made disjointly and via heuristic means. A significant obstacle in adopting more effective heuristic or AI-driven solutions is the absence of a quick and reliable introspective dynamic model to evaluate and refine alternative approaches. In this study, we aim to develop such an interactive system using real-world data. By examining job execution records from the PanDA workflow management system, we have pinpointed key performance indicators such as queuing time, error rate, and the extent of remote data access. The dataset includes five months of activity. Additionally, we are creating a generative AI model to simulate time series of payloads, which incorporate visible features like category, event count, and submitting group, as well as hidden features like the total computational load—derived from existing PanDA records and computing site capabilities. These hidden features, which are not visible to job allocators, whether heuristic or AI-driven, influence factors such as queuing times and data movement.

kilic, Ozgur Ozan [Brookhaven National Laboratory ↗
DOE OSTI2021

DOE JGI Metagenome Workflow

The DOE Joint Genome Institute (JGI) Metagenome Workflow performs metagenome data processing, including assembly; structural, functional, and taxonomic annotation; and binning of metagenomic data sets that are subsequently included into the Integrated Microbial Genomes and Microbiomes (IMG/M) (I.-M. A. Chen, K. Chu, K. Palaniappan, A. Ratner, et al., Nucleic Acids Res, 49:D751–D763, 2021, https://doi.org/10.1093/nar/gkaa939) comparative analysis system and provided for download via the JGI data portal (https://genome.jgi.doe.gov/portal/). This workflow scales to run on thousands of metagenome samples per year, which can vary by the complexity of microbial communities and sequencing depth. Here, we describe the different tools, databases, and parameters used at different steps of the workflow to help with the interpretation of metagenome data available in IMG and to enable researchers to apply this workflow to their own data. We use 20 publicly available sediment metagenomes to illustrate the computing requirements for the different steps and highlight the typical results of data processing. The workflow modules for read filtering and metagenome assembly are available as a workflow description language (WDL) file (https://code.jgi.doe.gov/BFoster/jgi_meta_wdl). The workflow modules for annotation and binning are provided as a service to the user community at https://img.jgi.doe.gov/submit and require filling out the project and associated metadata descriptions in the Genomes OnLine Database (GOLD) (S. Mukherjee, D. Stamatis, J. Bertsch, G. Ovchinnikova, et al., Nucleic Acids Res, 49:D723–D733, 2021, https://doi.org/10.1093/nar/gkaa983).

59 BASIC BIOLOGICAL SCIENCES↗