Search NASA⌕ Search

SEARCH · Search NASA

Results for “data quality issues”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Online Tools for Uncovering Data Quality (DQ) Issues in Satellite-Based Global Precipitation Products

Data quality (DQ) has many attributes or facets (i.e., errors, biases, systematic differences, uncertainties, benchmark, false trends, false alarm ratio, etc.)Sources can be complicated (measurements, environmental conditions, surface types, algorithms, etc.) and difficult to be identified especially for multi-sensor and multi-satellite products with bias correction (TMPA, IMERG, etc.) How to obtain DQ info fast and easily, especially quantified info in ROI Existing parameters (random error), literature, DIY, etc.How to apply the knowledge in research and applications.Here, we focus on online systems for integration of products and parameters, visualization and analysis as well as investigation and extraction of DQ information.

precipitation↗

SeaWiFS Postlaunch Calibration and Validation Analyses

The effort to resolve data quality issues and improve on the initial data evaluation methodologies of the SeaWiFS Project was an extensive one. These evaluations have resulted, to date, in three major reprocessings of the entire data set where each reprocessing addressed the data quality issues that could be identified up to the time of the reprocessing. Three volumes of the SeaWiFS Postlaunch Technical Report Series (Volumes 9, 10, and 11) are needed to document the improvements implemented since launch. Volume 10 continues the sequential presentation of postlaunch data analysis and algorithm descriptions begun in Volume 9. Chapter 1 of Volume 10 describes an absorbing aerosol index, similar to that produced by the Total Ozone Mapping Spectrometer (TOMS) Project, which is used to flag pixels contaminated by absorbing aerosols, such as, dust and smoke. Chapter 2 discusses the algorithm being used to remove SeaWiFS out-of-band radiance from the water-leaving radiances. Chapter 3 provides an itemization of all significant changes in the processing algorithms for each of the first three reprocessings. Chapter 4 shows the time series of global clear water and deep-water (depths greater than 1,000m) bio-optical and atmospheric properties (normalized water-leaving radiances, chlorophyll, atmospheric optical depth, etc.) based on the eight-day composites as a check on the sensor calibration stability. Chapter 5 examines the variation in the derived products with scan angle using high resolution data around Hawaii to test for residual scan modulation effects and atmospheric correction biases. Chapter 6 provides a methodology for evaluating the atmospheric correction algorithm and atmospheric derived products using ground-based observations. Similarly, Chapter 7 presents match-up comparisons of coincident satellite and in situ data to determine the accuracy of the water-leaving radiances, chlorophyll a, and K(490) products.

Hooker, Stanford B.↗

SeaWiFS Postlaunch Calibration and Validation Analyses

The effort to resolve data quality issues and improve on the initial data evaluation methodologies of the Sea-viewing Wide Field-of-view Sensor (SeaWiFS) Project was an extensive one. These evaluations have resulted, to date, in three major reprocessings of the entire data set where each reprocessing addressed the data quality issues that could be identified up to the time of each reprocessing. The number of chapters (21) needed to document this extensive work in the SeaWiFS Postlaunch Technical Report Series requires three volumes. The chapters in Volumes 9, 10, and 11 are in a logical order sequencing through sensor calibration, atmospheric correction, masks and flags, product evaluations, and bio-optical algorithms. The first chapter of Volume 9 is an overview of the calibration and validation program, including a table of activities from the inception of the SeaWiFS Project. Chapter 2 describes the fine adjustments of sensor detector knee radiances, i.e., radiance levels where three of the four detectors in each SeaWiFS band saturate. Chapters 3 and 4 describe the analyses of the lunar and solar calibration time series, respectively, which are used to track the temporal changes in radiometric sensitivity in each band. Chapter 5 outlines the procedure used to adjust band 7 relative to band 8 to derive reasonable aerosol radiances in band 7 as compared to those in band 8 in the vicinity of Lanai, Hawaii, the vicarious calibration site. Chapter 6 presents the procedure used to estimate the vicarious calibration gain adjustment factors for bands 1-6 using the waterleaving radiances from the Marine Optical Buoy (MOBY) offshore of Lanai. Chapter 7 provides the adjustments to the coccolithophore flag algorithm which were required for improved performance over the prelaunch version. Chapter 8 is an overview of the numerous modifications to the atmospheric correction algorithm that have been implemented. Chapter 9 describes the methodology used to remove artifacts of sun glint contamination for portions of the imagery outside the sun glint mask. Finally, Chapter 10 explains a modification to the ozone interpolation method to account for actual time differences between the SeaWiFS and Total Ozone Mapping Spectrometer (TOMS) orbits.

Hooker, Stanford B.↗

Analysis of ETMS Data Quality for Traffic Flow Management Decisions

The data needed for air traffic flow management decision support tools is provided by the Enhanced Traffic Management System (ETMS). This includes both the tools that are in current use and the ones being developed for future deployment. Since the quality of decision support provided by all these tools will be influenced by the quality of the input ETMS data, an assessment of ETMS data quality is needed. Motivated by this desire, ETMS data quality is examined in this paper in terms of the unavailability of flight plans, deviation from the filed flight plans, departure delays, altitude errors and track data drops. Although many of these data quality issues are not new, little is known about their extent. A goal of this paper is to document the magnitude of data quality issues supported by numerical analysis of ETMS data. Guided by this goal, ETMS data for a 24-hour period were processed to determine the number of aircraft with missing flight plan messages at any given instant of time. Results are presented for aircraft above 18,000 feet altitude and also at all altitudes. Since deviation from filed flight plan is also a major cause of trajectory-modeling errors, statistics of deviations are presented. Errors in proposed departure times and ETMS-generated vertical profiles are also shown. A method for conditioning the vertical profiles for improving demand prediction accuracy is described. Graphs of actual sector counts obtained using these vertical profiles are compared with those obtained using the Host data for sectors in the Fort Worth Center to demonstrate the benefit of preprocessing. Finally, results are presented to quantify the extent of data drops. A method for propagating track positions during ETMS data drops is also described.

Chatterji, Gano B.↗

Regional scale hydrology with a new land surface processes model

Through the CaPE Hydrometeorology Project, we have developed an understanding of some of the unique data quality issues involved in assimilating data of disparate types for regional-scale hydrologic modeling within a GIS framework. Among others, the issues addressed here include the development of adequate validation of the surface water budget, implementation of the STATSGO soil data set, and implementation of a remote sensing-derived landcover data set to account for surface heterogeneity. A model of land surface processes has been developed and used in studies of the sensitivity of surface fluxes and runoff to soil and landcover characterization. Results of these experiments have raised many questions about how to treat the scale-dependence of land surface-atmosphere interactions on spatial and temporal variability. In light of these questions, additional modifications are being considered for the Marshall Land Surface Processes Model. It is anticipated that these techniques can be tested and applied in conjunction with GCIP activities over regional scales.

Laymon, Charles↗

Data Science and the Knowledge Discovery Adventure

This talk will cover the important steps involved in the data science and knowledge discovery process: • Initial fact gathering (interview domain experts, review reports, articles, state-of-the-art) • Identify the problem (prediction, classification, statistical analysis, etc.) • Survey supporting data sources • Understand the data (numerical, categorical, text, sampling rate, data quality issues, etc.) • Selecting relevant features and sources • Acquire the data (set up agreements with the data stewards, APIs to download, etc.) • Merge data sources (temporal, spatial, common key, other ontologies...) • Feature Engineering (non linear domain knowledge or physics-based relationships) • Build data processing pipeline (may need to tap into data stream, develop parallel processing algorithm, federated learning etc.) • Build model and test (tune hyper-parameters, cross validation.) • Analyze/Validate results (do the results make sense. Does it answer the original question). • Deploy/Publish (Monitor and assess benefits)

Data science↗

Navigation Algorithms for the SeaWiFS Mission

The navigation algorithms for the Sea-viewing Wide Field-of-view Sensor (SeaWiFS) were designed to meet the requirement of 1-pixel accuracy-a standard deviation (sigma) of 2. The objective has been to extract the best possible accuracy from the spacecraft telemetry and avoid the need for costly manual renavigation or geometric rectification. The requirement is addressed by postprocessing of both the Global Positioning System (GPS) receiver and Attitude Control System (ACS) data in the spacecraft telemetry stream. The navigation algorithms described are separated into four areas: orbit processing, attitude sensor processing, attitude determination, and final navigation processing. There has been substantial modification during the mission of the attitude determination and attitude sensor processing algorithms. For the former, the basic approach was completely changed during the first year of the mission, from a single-frame deterministic method to a Kalman smoother. This was done for several reasons: a) to improve the overall accuracy of the attitude determination, particularly near the sub-solar point; b) to reduce discontinuities; c) to support the single-ACS-string spacecraft operation that was started after the first mission year, which causes gaps in attitude sensor coverage; and d) to handle data quality problems (which became evident after launch) in the direct-broadcast data. The changes to the attitude sensor processing algorithms primarily involved the development of a model for the Earth horizon height, also needed for single-string operation; the incorporation of improved sensor calibration data; and improved data quality checking and smoothing to handle the data quality issues. The attitude sensor alignments have also been revised multiple times, generally in conjunction with the other changes. The orbit and final navigation processing algorithms have remained largely unchanged during the mission, aside from refinements to data quality checking. Although further improvements are certainly possible, future evolution of the algorithms is expected to be limited to refinements of the methods presented here, and no substantial changes are anticipated.

Hooker, Stanford B.↗

Detection and Use of Load and Gage Output Repeats of Wind Tunnel Strain-Gage Balance Data

Criteria are discussed that may be used for the detection of load and gage output repeats of wind tunnel strain-gage balance data. First, empirical thresholds are introduced that help determine if the loads or electrical outputs of a pair of balance calibration or check load data points match. A threshold of 0.01 percent of the load capacity is suggested for the identification of matching loads. Similarly, a threshold of 0.1 microV/V is recommended for the identification of matching electrical outputs. Two examples for the use of load and output repeats are discussed to illustrate benefits of the implementation of a repeat point detection algorithm in a balance data analysis software package. The first example uses the suggested load threshold to identify repeat data points that may be used to compute pure errors of the balance loads. This type of analysis may reveal hidden data quality issues that could potentially be avoided by making calibration process improvements. The second example uses the electrical output threshold for the identification of balance fouling. Data from the calibration of a six-component force balance is used to illustrate the calculation of the pure error of the balance loads.

strain-gage balance↗

Core Community Specifications for Electron Microprobe Operating Systems: Software, Quality Control, and Data Management Issues

Modem electron microprobe systems have become increasingly sophisticated. These systems utilize either UNIX or PC computer systems for measurement, automation, and data reduction. These systems have undergone major improvements in processing, storage, display, and communications, due to increased capabilities of hardware and software. Instrument specifications are typically utilized at the time of purchase and concentrate on hardware performance. The microanalysis community includes analysts, researchers, software developers, and manufacturers, who could benefit from exchange of ideas and the ultimate development of core community specifications (CCS) for hardware and software components of microprobe instrumentation and operating systems.

Fournelle, John↗

A Quality Screening Service for Remote Sensing Data

NASA provides a wide variety of Earth-observing satellite data products to a diverse community. These data are annotated with quality information in a variety of ways, with the result that many users struggle to understand how to properly account for quality when dealing with satellite data. To address this issue, a Data Quality Screening Service (DQSS) is being implemented for a number of datasets. The DQSS will enable users to obtain data files in which low-quality pixels have been filtered out, based either on quality criteria recommended by the science team or on the user s particular quality criteria. The objective is to increase proper utilization of this critical quality data in science data analysis of satellite data products.

Lynnes, Christopher↗

Supporting Global Air Quality Management Needs With A Flexible Data Fusion Tool for Estimation and Forecasting in Google Earth Engine

High spatial and temporal resolution air quality estimation and forecasting can be enhanced by combining global data sources, like chemical transport models and satellite remote sensing, with local information from regulatory and low-cost air quality monitors. Successful integration of data from these diverse sources is complicated by many factors, however, including differences in spatial and temporal resolution, data availability and latency issues, varying data quality, and large computational and data storage requirements. This presentation will provide an overview of a NASA-funded effort to develop the foundation for future operationalization of air quality forecasting for world-wide end-users and integration into their air quality management decision processes, which will be achieved in future phases of this multi-year project. We will summarize our progress in developing a data fusion system using the Google Earth Engine platform which can integrate model, satellite, and surface-level monitoring datasets to enhance estimation and forecasting of air-quality-relevant pollutants at sub-daily and sub-city scales. The tool is being developed in close cooperation with several city- and regional-level air quality managers in the USA and around the world. Our end-goal is to provide these air quality managers with the information they need to assess and anticipate the impacts of poor air quality, track changes in air quality due to ongoing mitigation efforts and land use changes, and identify ways to improve their air quality monitoring strategies. This presentation will focus on recent advances achieved through the project, including integration of multiple air quality datasets in a prototype data fusion system in Google Earth Engine, the quantification of uncertainties associated with our data fusion approach, and the development of user interfaces and visualization tools to convey air quality information in a way which best meets end-user needs.

Carl Malings↗

Supporting Global Air Quality Management Needs With A Flexible Data Fusion Tool for Estimation and Forecasting in Google Earth Engine

High spatial and temporal resolution air quality estimation and forecasting can be enhanced by combining global data sources, like chemical transport models and satellite remote sensing, with local information from regulatory and low-cost air quality monitors. Successful integration of data from these diverse sources is complicated by many factors, however, including differences in spatial and temporal resolution, data availability and latency issues, varying data quality, and large computational and data storage requirements. This presentation will provide an overview of a NASA-funded effort to develop the foundation for future operationalization of air quality forecasting for world-wide end-users and integration into their air quality management decision processes, which will be achieved in future phases of this multi-year project. We will summarize our progress in developing a data fusion system using the Google Earth Engine platform which can integrate model, satellite, and surface-level monitoring datasets to enhance estimation and forecasting of air-quality-relevant pollutants at sub-daily and sub-city scales. The tool is being developed in close cooperation with several city- and regional-level air quality managers in the USA and around the world. Our end-goal is to provide these air quality managers with the information they need to assess and anticipate the impacts of poor air quality, track changes in air quality due to ongoing mitigation efforts and land use changes, and identify ways to improve their air quality monitoring strategies. This presentation will focus on recent advances achieved through the project, including integration of multiple air quality datasets in a prototype data fusion system in Google Earth Engine, the quantification of uncertainties associated with our data fusion approach, and the development of user interfaces and visualization tools to convey air quality information in a way which best meets end-user needs.

Carl Malings↗

Increasing Efficiency at the NTF by Optimizing Model AoA Positioning

The National Transonic Facility (NTF) at NASA Langley Research Center (LaRC) is a national resource for aeronautical research and development. The government, military and private industries rely on the capability of this facility for realistic flight data. Reducing the operation costs and keeping the NTF affordable is essential for aeronautics research. The NTF is undertaking an effort to reduce the time between data points during a pitch polar. This reduction is being driven by the operating costs of a cryogenic facility. If the time per data point can be reduced, a substantial cost savings can be realized from a reduction in liquid nitrogen (LN2) consumption. It is known that angle-of-attack (AoA) positioning is the longest lead-time item between points. In January 2005 a test was conducted at the NTF to determine the cause of the long lead-time so that an effort could be made to improve efficiency. The AoA signal at the NTF originates from onboard instrumentation then travels through a number of different systems including the signal conditioner, digital voltmeter, and the data system where the AoA angle is calculated. It is then fed into a closed loop control system that sets the model position. Each process along this path adds to the time per data point affecting the efficiency of the data taking process. Due to the nature of the closed loop feed back AoA control and the signal path, it takes approximately 18 seconds to take one pitch pause point with a typical AoA increment. Options are being investigated to reduce the time delay between points by modifying the signal path. These options include: reduced signal filtering, using analog channels instead of a digital volt meter (DVM), re-routing the signal directly to the AoA control computer and implementing new control algorithms. Each of these has potential to reduce the positioning time and together the savings could be significant. These timesaving efforts are essential but must be weighed against possible loss of data quality. For example, a reduction in filtering can introduce noise into the signal and using analog channels could result in some loss of accuracy. Data quality assessments need to be performed concurrently with timesaving techniques since data quality parameters are essential in maintaining facility integrity. This paper will highlight time saving efforts being undertaken or studied at the NTF. It will outline the instrumentation and computer systems involved in setting of the model pitch attitude then suggest changes to the process and discuss how these system changes would effect the time between data points. It also discusses the issue of data quality and how the potential efficiency changes in the system could affect it. Lastly, it will discuss the possibility of using an open loop control system and give some pros and cons of this method.

Crawford, Bradley L.↗

Ambiguity of Quality in Remote Sensing Data

This slide presentation reviews some of the issues in quality of remote sensing data. Data "quality" is used in several different contexts in remote sensing data, with quite different meanings. At the pixel level, quality typically refers to a quality control process exercised by the processing algorithm, not an explicit declaration of accuracy or precision. File level quality is usually a statistical summary of the pixel-level quality but is of doubtful use for scenes covering large areal extents. Quality at the dataset or product level, on the other hand, usually refers to how accurately the dataset is believed to represent the physical quantities it purports to measure. This assessment often bears but an indirect relationship at best to pixel level quality. In addition to ambiguity at different levels of granularity, ambiguity is endemic within levels. Pixel-level quality terms vary widely, as do recommendations for use of these flags. At the dataset/product level, quality for low-resolution gridded products is often extrapolated from validation campaigns using high spatial resolution swath data, a suspect practice at best. Making use of quality at all levels is complicated by the dependence on application needs. We will present examples of the various meanings of quality in remote sensing data and possible ways forward toward a more unified and usable quality framework.

Lynnes, Christopher↗

Quality Control Methods of Tower Data at Kennedy Space Center’s Launch Complex 39-B and the USAF Cape Canaveral 500 Foot Tower

The National Aeronautics and Space Administration (NASA) has long used meteorological data from weather towers located at Kennedy Space Center (KSC) and the United States Air Force’s Eastern Range(ER) in support of their various launch vehicles and climate studies. Some of the most valuable data is gathered at the Launch Complex 39-B (LC39-B) and the 500 foot tower located approximately two miles west of LC39-B and three miles north of the Vehicle Assembly Building (VAB). The data from LC39-B gives NASA Space Launch System (SLS) engineers valuable insight into the weather the vehicle can expect at the pad, while the 500 foot tower has a longer period of record and has been used by the Shuttle and SLS programs. However, numerous data quality (DQ) control issues have arisen when utilizing this data for analysis. In addition, instrumentation tends to deteriorate faster than normal due to the corrosive nature of the high salt content in the air. NASA’s Natural Environments branch has developed several QC databases of these towers, but these studies were for fixed periods of records. EV44 identified a need for a continually updated and QC’ed database of tower data not only to provide along term QC’ed database for vehicle and climate analyses, but also to provide the capability to investigate recent weather events such as downbursts and other high wind events. This study follows the methods developed in prior Natural Environment QC tower databases, which includes variable specific QC thresholds and checks used to generate QC flags. However, this study developed new techniques such as the development and design of the graphical user interface (GUI) for manual verification, and the creation of the final monthly QC files.

Quality Control↗

Rainfall and Snowfall Observations by the Airborne Dual-frequency Precipitation Radar during the Wakasa Bay Experiment

Radar data obtained through the NASA/JPL Airborne Precipitation Radar APR-2 during the Wakasa Bay Experiment in January/February 2003 were processed to obtain calibrated reflectivity measurements, rainfall/snowfall velocity measurements, classification of the surface type and detection of the boundaries of the melting layer of precipitation. In this paper the processing approach is described together with an overview of the resulting data quality and known issues.

field campaign↗

Preliminary Assessment of JERS-1 SAR to Discriminating Boreal Landscape Features for the Boreal Forest Mapping Project

This paper presents an overview of the JERS-1 North American Boreal Forest Mapping Project and a preliminary assessment of JERS-1 SAR imagery for application to discriminating features applicable to boreal landscape processes. The present focus of the JERS-1 North American Boreal Forest Mapping Project is the production of continental scale wintertime and summertime SAR mosaics of the North American boreal forest for distribution to the science community. As part of this effort, JERS-1 imagery has been collected over much of Alaska and Canada during the 1997-98 winter and 1998 summer seasons. To complete the mosaics, these data will be augmented with data collected during previous years. These data will be made available to the scientific community via CD ROM containing these and similar data sets compiled from companion studies of Asia and Europe. Regional landscape classification with SAR is important for the baseline information it will provide about distribution of woodlands, positions of treeline, current forest biomass, distribution of wetlands, and extent of major rivercourses. As well as setting the stage for longer term change detection, comparisons across several years provides additional baseline information about short-term landscape change. Rapid changes, including those driven by fire, permafrost heat balance, flooding, and insect outbreaks can dominate boreal systems. We examine JERS-1 imagery covering selected sites in Alaska and Canada to assess quality and applicability to such relevant ecological and hydrological issues. The data are generally of high quality and illustrate many potential applications. A texture-based classification scheme is applied to selected regions to assess the applicability of these data for distinguishing distribution of such landcover types as wetland, tundra, woodland and forested landscapes.

McDonald, Kyle↗