Search NASA⌕ Search

SEARCH · Search NASA

Results for “data processing steps”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Development of advanced techniques for rotorcraft state estimation and parameter identification

An integrated methodology for rotorcraft system identification consists of rotorcraft mathematical modeling, three distinct data processing steps, and a technique for designing inputs to improve the identifiability of the data. These elements are as follows: (1) a Kalman filter smoother algorithm which estimates states and sensor errors from error corrupted data. Gust time histories and statistics may also be estimated; (2) a model structure estimation algorithm for isolating a model which adequately explains the data; (3) a maximum likelihood algorithm for estimating the parameters and estimates for the variance of these estimates; and (4) an input design algorithm, based on a maximum likelihood approach, which provides inputs to improve the accuracy of parameter estimates. Each step is discussed with examples to both flight and simulated data cases.

Hall, W. E., Jr.↗

The Rings Node for the Planetary Data System

The Planetary Data System's Rings Node is devoted to the archiving and distributing of scientific data sets relevant to planetary ring systems. The two major classes of ring data are images and occultation profiles, although a variety of additional data types (e.g. spectra, particle absorption signatures, etc.) are also of interest. A large fraction of our data sets are from the Voyager missions to the outer planets, but Earth-based and Hubble Space Telescope data sets are also represented. Archiving work often includes re-formatting the data into standardized formats and reconstructing some of the data processing steps. The Rings Node also performs a variety of services to support research into these data sets. These services include developing on-line catalogs and information systems, filling orders for data, developing software tools, and coordinating special observing campaigns.

Showalter, Mark R.↗

Analysis of High-Throughput ELISA Microarray Data

Our research group develops analytical methods and software for the high-throughput analysis of quantitative enzyme-linked immunosorbent assay (ELISA) microarrays. ELISA microarrays differ from DNA microarrays in several fundamental aspects and most algorithms for analysis of DNA microarray data are not applicable to ELISA microarrays. In this review, we provide an overview of the steps involved in ELISA microarray data analysis and how the statistically sound algorithms we have developed provide an integrated software suite to address the needs of each data-processing step. The algorithms discussed are available in a set of open-source software tools (http://www.pnl.gov/statistics/ProMAT).

59 BASIC BIOLOGICAL SCIENCES↗

2018 NISAR Applications Workshop: Forest and Disturbance; Workshop Report

Forest lands cover the globe and are important sources for providing ecosystem services including: carbon sequestration, biodiversity, timber, air and water quality. As such, counties around the world have dedicated programs for managing them. Accurate and timely information concerning the status of these forests (moisture, biomass, disturbance type, etc.) is essential to those Nations’ human and ecological health as well as economy. The joint NASA/US Forest Service workshop focused on arming forest land managers with observations and remote sensing information from the upcoming NASA-ISRO (Indian Space Research Organization) SAR (Synthetic Aperture Radar) (NISAR) satellite mission (expected to launch early 2022). Participants included representatives from different US Federal Agencies, private sector, and non-governmental organizations (NGO) that are key players in facilitating integration of Earth Observations (EO) into forest management and decision support workflows. They included scientists, technicians, and program managers with a responsibility for data acquisition and exploitation such as product development, delivery, and use, and capacity building. Discussions were held over two days to convey the broader forest and disturbance community information needs for various representative participants and programs and to facilitate the delivery of NISAR mission geospatial products and observational capabilities. Case studies were presented to demonstrate the current state of practice in the use of SAR remote sensing for applications of direct importance to forest and disturbance land management community. Eleven organizations presented their information requirements in response to a set of questions provided by the NASA team, then the NASA team responded by describing the degree to which NISAR could meet these requirements. Discussion ensued about needed data product specifications to increase utility (e.g., projection, latency, etc.), tools and capacity building. The general findings of this workshop were that (a) NISAR observations will be particularly useful to the global forest carbon and disturbance monitoring applications, but that certain data product design decisions (projections and radiometric and terrain corrections) need to be considered to increase utility; b) the biomass and disturbance detection algorithms meet many of the community needs, however there are other information products of value (e.g., soil moisture or disturbance classification, not just detection) and all products should be compliant with existing community standards for reporting uncertainty; c) providing SAR education to the community will be key specifically thinking about putting the information first and the SAR theory second, providing a simple guide of standard data processing steps (e.g., dB (decibel) to power conversion and speckle filtering); d) the community needs a user-friendly interface for finding free, archived data over their geographic regions of interest; e) user-friendly tools that connect to open-sources GIS (Global Information System) software (e.g., QGIS (Quantum GIS)) that include a graphical user interface (GUI) for SAR processing that enables both download and cloud processing. To integrate these findings and prepare the community before NISAR launches, it was suggested that there be a dedicated NISAR Forest and Disturbance Applications Working Group (as per the specifications in the NISAR Utilization Plan). After launch, it was decided that the community continue capacity building activities.

Stavros, Natasha↗

A characterization of workflow management systems for extreme-scale applications

We present that the automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today’s computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. Finally, the paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

97 MATHEMATICS AND COMPUTING↗

Automated Precision Orbit Determination for TOPEX/Poseidon with GPS

A highly automated GPS data processing system for the orbit determination of TOPEX/Poseidon is described. The orbit is recovered to an estimated accuracy of better than 4 cm in altitude, 6 cm crosstrack, and 11 cm down track. The RMS postfit residuals on the ionospherically calibrated carrier phase observable are less than 5 mm. The RMS difference over a 4.5-hour overlap period between two 30-hour data arcs is 1 cm in altitude, 5 cm cross track, and 4 cm down track. These results can be obtained within two days of onboard GPS data collection. Most of the data processing for a 30-hour arc of GPS data can be performed on a single workstation in less than 6 hours of CPU time. The estimation scenarios are explained, the automated data processing steps are described, and means to assess solution quality are discussed.

automation GPS data processing orbit determination↗

DEIMoS GUI: An Open-Source User Interface for a High-Dimensional Mass Spectrometry Data Processing Tool

In this paper, we report the creation of a graphical user interface (GUI) for the Data Extraction for Integrated Multidimensional Spectrometry (DEIMoS) tool. DEIMoS is a Python package to process data from high-dimensional mass spectrometry measurements. It is divided into several modules, each representing a data processing step, such as peak detection, alignment, and tandem mass spectra extraction and deconvolution. The inputs for and outputs from DEIMoS can include millions of N-dimensional data points, which can be challenging to visualize in a way that is interactive, informative, and responsive. Here, we used the HoloViz Python data stack, including DataShader and Param, to create an interactive visualization of mass spectrometry data. We believe the GUI will increase the accessibility of DEIMoS, and the visualization methods could be useful for other open-source mass spectrometry tools.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Compensation of Oxygen Transmittance Effects for Proximal Sensing Retrieval of Canopy-Leaving Sun-Induced Chlorophyll Fluorescence

Estimates of Sun-Induced vegetation chlorophyll Fluorescence (SIF) using remote sensing techniques are commonly determined by exploiting solar and/or telluric absorption features. When SIF is retrieved in the strong oxygen (O2) absorption features, atmospheric effects must always be compensated. Whereas correction of atmospheric effects is a standard airborne or satellite data processing step, there is no consensus regarding whether it is required for SIF proximal-sensing measurements nor what is the best strategy to be followed. Thus, by using simulated data, this work provides a comprehensive analysis about how atmospheric effects impact SIF estimations on proximal sensing, regarding: (1) the sensor height above the vegetated canopy; (2) the SIF retrieval technique used, e.g., Fraunhofer Line Discriminator (FLD) family or Spectral Fitting Methods (SFM); and (3) the instrument's spectral resolution. We demonstrate that for proximal-sensing scenarios compensating for atmospheric effects by simply introducing the O2 transmittance function into the FLD or SFM formulations improves SIF estimations. However, these simplistic corrections still lead to inaccurate SIF estimations due to the multiplication of spectrally convolved atmospheric transfer functions with absorption features. Consequently, a more rigorous oxygen compensation strategy is proposed and assessed by following a classic airborne atmospheric correction scheme adapted to proximal sensing. This approach allows compensating for the O2 absorption effects and, at the same time, convolving the high spectral resolution data according to the corresponding Instrumental Spectral Response Function (ISRF) through the use of an atmospheric radiative transfer model. Finally, due to the key role of O2 absorption on the evaluated proximal-sensing SIF retrieval strategies, its dependency on surface pressure (p) and air temperature (T) was also assessed. we combined simulated spectral data with p and T measurements obtained for a one-year period in the Hyytiala Forestry Field Station in Finland. Of importance hereby is that seasonal dynamics in terms of T and p, if not appropriately considered as part of the retrieval strategy, can result in erroneous SIF seasonal trends that mimic those of known dynamics for temperature-dependent physiological responses of vegetation.

"sun–induced chlorophyll fluorescence (SIF); pro↗

Data and scripts from: “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”

This data package includes data and scripts from the manuscript “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”.The study addressed common challenges faced in environmental sensing and modeling, including uncertain input data, missing sensor observations, and high-dimensional datasets with interrelated but redundant variables. Point-scaled meteorological and soil sensor observations were perturbed with noises and missing values, and denoising autoencoder (DAE) neural networks were developed to reconstruct the perturbed data and further predict evapotranspiration. This study concluded that (1) the reconstruction quality of each variable depends on its cross-correlation and alignment to the underlying data structure, (2) uncertainties from the models were overall stronger than those from the data corruption, and (3) there was a tradeoff between reducing bias and reducing variance when evaluating the uncertainty of the machine learning models.This package includes:(1) Four ipython scripts (.ipynb): “DAE_train.ipynb” trains and evaluates DAE neural networks, “DAE_predict.ipynb” makes predictions from the trained DAE models, “ET_train.ipynb” trains and evaluates ET prediction neural networks, and “ET_predict.ipynb” makes predictions from trained ET models.(2) One python file (.py): “methods.py” includes all user-defined functions and python codes used in the ipython scripts.(3) A “sub_models” folder that includes five trained DAE neural networks (in pytorch format, .pt), which could be used to ingest input data before being fed to the downstream ET models in ‘ET_train.ipynb” or ‘ET_predict.ipynb’.(4) Two data files (.csv). Daily meteorological, vegetation, and soil data is in “df_data.csv”, where “df_meta.csv” contains the location and time information of “df_data.csv”. Each row (index) in “df_meta.csv” corresponds to each row in “df_data.csv”. These data files are formatted to follow the data structure requirements and be directly used in the ipython scripts, and they have been shuffled chronologically to train machine learning models. The meteorological and soil data was collected using point sensors between 2019-2023 at(4.a) Three shrub-dominated field sites in East River, Colorado (named “ph1”, “ph2” and “sg5” in “df_meta.csv”, where “ph1” and “ph2” were located at PumpHouse Hillslopes, and “sg5” was at Snodgrass Mountain meadow) and(4.b) One outdoor, mesoscale, and herbaceous-dominated experiment in Berkeley, California (named “tb” in “df_meta.csv”, short for Smartsoils Testbed at Lawrence Berkeley National Lab).- See "df_data_dd.csv" and "df_meta_dd.csv" for variable descriptions and the Methods section for additional data processing steps. See "flmd.csv" and "README.txt" for brief file descriptions.- All ipython scripts and python files are written in and require PYTHON language software.

54 ENVIRONMENTAL SCIENCES↗

IOTA : integration optimization, triage and analysis tool for the processing of XFEL diffraction images

Serial femtosecond crystallography (SFX) uses an X-ray free-electron laser to extract diffraction data from crystals not amenable to conventional X-ray light sources owing to their small size or radiation sensitivity. However, a limitation of SFX is the high variability of the diffraction images that are obtained. As a result, it is often difficult to determine optimal indexing and integration parameters for the individual diffraction images. Presented here is a software package, called IOTA , which uses a grid-search technique to determine optimal spot-finding parameters that can in turn affect the success of indexing and the quality of integration on an image-by-image basis. Integration results can be filtered using a priori information about the Bravais lattice and unit-cell dimensions and analyzed for unit-cell isomorphism, facilitating an improvement in subsequent data-processing steps.

Chemistry↗

Qubit noise deconvolution

We present a noise deconvolution technique to remove a wide class of noises when performing arbitrary measurements on qubit systems. In particular, we derive the inverse map of the most common single qubit noisy channels, and exploit it at the data processing step to obtain noise-free estimates of observables evaluated on a qubit system subject to known noise. We illustrate a self-consistency check to ensure that the noise characterization is accurate providing simulation results for the deconvolution of a generic Pauli channel, as well as experimental evidence of the deconvolution of decoherence noise occurring on Rigetti quantum hardware.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Automated analysis of unlabeled PV data with Solar Data Tools software: Overview and feature updates

Distributed rooftop PV systems: ubiquitous, yet commonly have unlabeled data Difficult or impossible to form a performance index We developed Solar Data Tools (SDT), an open-source Python library for analyzing PV power (and irradiance) time-series data SDT enables analysis of unlabeled PV data—no model, no meteorological data, no performance index required Takes a statistical signal processing approach Data processing steps are largely pre-defined and automatic regardless of system type—from utility tracking systems to multi-pitch rooftop systems

Meyers-Im, Bennet E↗

DEIMoS: An Open-Source Tool for Processing High-Dimensional Mass Spectrometry Data

We present DEIMoS: Data Extraction for Integrated Multidimensional Spectrometry, a Python application programming interface and command-line tool for high-dimensional mass spectrometry data analysis workflows, offering ease of development and access to efficient algorithmic implementations. Functionality includes feature detection, feature alignment, collision cross section calibration, isotope detection, and MS/MS spectral deconvolution, with the output comprising detected features aligned across study samples and characterized by mass, CCS, tandem mass spectra, and isotopic signature. Notably, DEIMoS operates on N-dimensional data, largely agnostic to acquisition instrumentation: algorithm implementations utilize all dimensions simultaneously to (i) offer greater separation between features, improving detection sensitivity, (ii) increase alignment/feature matching confidence among datasets, and (iii) mitigate convolution artifacts in tandem mass spectra. We demonstrate DEIMoS with LC-IMS-MS/MS data, demonstrating the advantages of a multidimensional approach in each data processing step.

Colby, Sean M↗

Ultra-high Information-content Chemical Imaging with Broadband Coherent Anti-Stokes Raman and Two-photon Fluorescence Lifetime Microscopy

Raman fingerprint spectroscopy and fluorescence lifetime imaging are emerging tools for studying metabolic profiles of biological specimens. While Raman fingerprint spectroscopy detects intrinsic molecular vibrations that reflect the molecular composition and chemical environment of a sample, fluorescence lifetime imaging measures changes in the excited-state lifetime of fluorophores that are sensitive to their microenvironments. Here, we present a multimodal imaging platform combining broadband coherent anti-Stokes Raman scattering (BCARS) and two-photon fluorescence lifetime imaging (2p-FLIM) microscopy that can acquire biologically relevant Raman fingerprint spectra and fluorescence lifetime signals in vivo and simultaneously. The tremendous chemical information obtained from spatially co-registered BCARS and 2p-FLIM images allows us to characterize the subtle differences between sub-cellular compartments and verify the potential false-positive results generated by fluorescence imaging alone. This is demonstrated by directly comparing the BCARS, 2p-FLIM, and two-photon excitation fluorescence(TPEF) signals simultaneously obtained from the same dye-stained organelle in the live, intact C. elegans expressing a green fluorescent protein (GFP) marker. In this work, we introduce the BCARS/2p-FLIM/TPEF setup scheme, the image acquisition steps, data processing, and representative results showing that the cross-modality imaging method enables rigorous characterization and in vivo detection at sub-cellular resolution. Furthermore, this protocol provides a framework for simultaneous chemical and fluorescence lifetime imaging to improve the accuracy of biological interpretation in complex living systems.

Xu, Haoyu [Georgia Institute of Technology, Atlant↗

A thermal emission spectrometer for field use

A field portable spectrometer has been built at the Jet Propulsion Laboratory which is suitable for collecting data relevant to remote sensing applications in the 4.7- to 5.6- and 8- to 12-micrometer atmospheric windows. The instrument employs a single cooled HgCdTe detector and a continuously variable filter wheel analyzer. The spectral range covered is 5- to 14.5-micrometers and the resolution is approximately 1.5 percent of the wavelength. A description of the hardware is followed by a discussion of the data processing steps leading to emissivity and radiance spectra. A section is devoted to the evaluation of the instrument performance with respect to spectral resolution, radiometric precision, and accuracy. Several examples of spectra acquired in the field are included.

Hoover, Gordon↗

A versatile system for processing geostationary satellite data with run-time visualization capability

To better predict global climate change, scientists are developing climate models that require interdisciplinary and collaborative efforts in their building. We are currently involved in several such projects but will briefly discuss activities in support of two such complementary projects: the Atmospheric Radiation Measurement (ARM) program of the Department of Energy and Sequoia 2000, a joint venture of the University of California, the private sector, and government agencies. Our contribution to the ARM program is to investigate the role of clouds on the top of the atmosphere and on surface radiance fields through the data analysis of surface and satellite observations and complex modeling of the interaction of radiation with clouds. One of our first ARM research activities involves the computation of the broadband shortwave surface irradiance from satellite observations. Geostationary satellite images centered over the first ARM observation site are received hourly over the Internet network and processed in real time to compute hourly and daily composite shortwave irradiance fields. The images and the results are transferred via a high-speed network to the Sequoia 2000 storage facility in Berkeley, where they are archived These satellite-derived results are compared with the surface observations to evaluate the accuracy of the satellite estimate and the spatial representation of the surface observations. In developing the software involved in calculating the surface shortwave irradiance, we have produced an environment whereby we can easily modify and monitor the data processing as required. Through the principles of modular programming, we have developed software that is easily modified as new algorithms for computation are developed or input data availability changes. In addition, the software was designed so that it could be run from an interactive, icon-driven, graphical interface, TCL-TK, developed by Sequoia 2000 participants. In this way, the data flow can be interactively assessed and altered as needed. In this environment, the intermediate data processing 'images' can be viewed, enabling the investigator to easily monitor the various data processing steps as they progress. Additionally, this environment allows the rapid testing of new processing modules and allows their effects to be visually compared with previous results.

Landsfeld, M.↗

A Long-Term and Reproducible Passive Microwave Sea Ice Concentration Data Record for Climate Studies and Monitoring

A long-term, consistent, and reproducible satellite-based passive microwave sea ice concentration climate data record (CDR) is available for climate studies, monitoring, and model validation with an initial operation capability (IOC). The daily and monthly sea ice concentration data are on the National Snow and Ice Data Center (NSIDC) polar stereographic grid with nominal 25 km × 25 km grid cells in both the Southern and Northern Hemisphere polar regions from 9 July 1987 to 31 December 2007. The data files are available in the NetCDF data format at http://nsidc.org/data/g02202.html and archived by the National Climatic Data Center (NCDC) of the National Oceanic and Atmospheric Administration (NOAA) under the satellite climate data record program (http://www.ncdc.noaa.gov/cdr/operationalcdrs.html). The description and basic characteristics of the NOAA/NSIDC passive microwave sea ice concentration CDR are presented here. The CDR provides similar spatial and temporal variability as the heritage products to the user communities with the additional documentation, traceability, and reproducibility that meet current standards and guidelines for climate data records. The data set, along with detailed data processing steps and error source information, can be found at http://dx.doi.org/10.7265/N5B56GN3.

monitoring↗

Autonomous landmark based spacecraft navigation system

An autonomous landmark based spacecraft navigation scheme is presented. This new schme involves the following data processing steps: image selection and planning; landmark detection; preliminary image matching; crater matching; database management; and finally landmark based orbit determination.

autonomous↗