Search NASA⌕ Search

SEARCH · Search NASA

Results for “data quality metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Parameter Sensitivity Study of the Wall Interference Correction System (WICS)

An off-line version of the Wall Interference Correction System (WICS) has been implemented for the "NASA Langley National Transonic Facility. The correction capability is currently restricted to corrections for solid wall interference in the model pitch plane for Mach numbers, less than 0.45 due to a limitation in tunnel calibration data. A study to assess output sensitivity to the aerodynamic parameters of Reynolds number and Mach number was conducted on this code to further ensure quality during the correction process. In addition, this paper includes all investigation into possible correction due to a semispan test technique using a non metric standoff and all improvement to the standard data rejection algorithm.

Walker, Eric L.↗

NASA Armstrong Flight Research Center Dynamics and Controls (530)

AFRC Controls and Dynamics Branch - Research Areas: - Traditional GN&C - Classical and advanced control algorithms - Risk based approaches for safety critical applications - Integration of novel sensors and sensor fusion (FOSS, LIDAR, etc.) - Trajectory optimization and control - Flight/System Dynamics - Equations of Motion and integrated modeling of vehicles and vehicle systems - Unique and novel vehicle dynamics - Methods for extracting relevant vehicle dynamic information from flight data - Human and vehicle interfaces and interactions - Handling Qualities and Pilot-in-the-loop oscillations (PIO) predictions and design metrics - Ride quality research - Autonomy - Novel algorithms, sensors, and sensor fusion - Bounding risk for testing automated systems - Pilot/Operator interactions - Engineering Support and Airworthiness Assessments - Aircraft modifications and experimental configurations

Chris Miller↗

Automated support for experience-based software management

To effectively manage a software development project, the software manager must have access to key information concerning a project's status. This information includes not only data relating to the project of interest, but also, the experience of past development efforts within the environment. This paper describes the concepts and functionality of a software management tool designed to provide this information. This tool, called the Software Management Environment (SME), enables the software manager to compare an ongoing development effort with previous efforts and with models of the 'typical' project within the environment, to predict future project status, to analyze a project's strengths and weaknesses, and to assess the project's quality. In order to provide these functions the tool utilizes a vast corporate memory that includes a data base of software metrics, a set of models and relationships that describe the software development environment, and a set of rules that capture other knowledge and experience of software managers within the environment. Integrating these major concepts into one software management tool, the SME is a model of the type of management tool needed for all software development organizations.

Valett, Jon D.↗

Integrating automated support for a software management cycle into the TAME system

Software managers are interested in the quantitative management of software quality, cost and progress. An integrated software management methodology, which can be applied throughout the software life cycle for any number purposes, is required. The TAME (Tailoring A Measurement Environment) methodology is based on the improvement paradigm and the goal/question/metric (GQM) paradigm. This methodology helps generate a software engineering process and measurement environment based on the project characteristics. The SQMAR (software quality measurement and assurance technology) is a software quality metric system and methodology applied to the development processes. It is based on the feed forward control principle. Quality target setting is carried out before the plan-do-check-action activities are performed. These methodologies are integrated to realize goal oriented measurement, process control and visual management. A metric setting procedure based on the GQM paradigm, a management system called the software management cycle (SMC), and its application to a case study based on NASA/SEL data are discussed. The expected effects of SMC are quality improvement, managerial cost reduction, accumulation and reuse of experience, and a highly visual management reporting system.

Sunazuka, Toshihiko↗

Automatic documentation system extension to multi-manufacturers' computers and to measure, improve, and predict software reliability

The DOMONIC system has been modified to run on the Univac 1108 and the CDC 6600 as well as the IBM 370 computer system. The DOMONIC monitor system has been implemented to gather data which can be used to optimize the DOMONIC system and to predict the reliability of software developed using DOMONIC. The areas of quality metrics, error characterization, program complexity, program testing, validation and verification are analyzed. A software reliability model for estimating program completion levels and one on which to base system acceptance have been developed. The DAVE system which performs flow analysis and error detection has been converted from the University of Colorado CDC 6400/6600 computer to the IBM 360/370 computer system for use with the DOMONIC system.

Simmons, D. B.↗

Methods of evaluating the effects of coding on SAR data

It is recognized that mean square error (MSE) is not a sufficient criterion for determining the acceptability of an image reconstructed from data that has been compressed and decompressed using an encoding algorithm. In the case of Synthetic Aperture Radar (SAR) data, it is also deemed to be insufficient to display the reconstructed image (and perhaps error image) alongside the original and make a (subjective) judgment as to the quality of the reconstructed data. In this paper we suggest a number of additional evaluation criteria which we feel should be included as evaluation metrics in SAR data encoding experiments. These criteria have been specifically chosen to provide a means of ensuring that the important information in the SAR data is preserved. The paper also presents the results of an investigation into the effects of coding on SAR data fidelity when the coding is applied in (1) the signal data domain, and (2) the image domain. An analysis of the results highlights the shortcomings of the MSE criterion, and shows which of the suggested additional criterion have been found to be most important.

Dutkiewicz, Melanie↗

Scatter-Reducing Sounding Filtration Using a Genetic Algorithm and Mean Monthly Standard Deviation

Retrieval algorithms like that used by the Orbiting Carbon Observatory (OCO)-2 mission generate massive quantities of data of varying quality and reliability. A computationally efficient, simple method of labeling problematic datapoints or predicting soundings that will fail is required for basic operation, given that only 6% of the retrieved data may be operationally processed. This method automatically obtains a filter designed to reduce scatter based on a small number of input features. Most machine-learning filter construction algorithms attempt to predict error in the CO2 value. By using a surrogate goal of Mean Monthly STDEV, the goal is to reduce the retrieved CO2 scatter rather than solving the harder problem of reducing CO2 error. This lends itself to improved interpretability and performance. This software reduces the scatter of retrieved CO2 values globally based on a minimum number of input features. It can be used as a prefilter to reduce the number of soundings requested, or as a post-filter to label data quality. The use of the MMS (Mean Monthly Standard deviation) provides a much cleaner, clearer filter than the standard ABS(CO2-truth) metrics previously employed by competitor methods. The software's main strength lies in a clearer (i.e., fewer features required) filter that more efficiently reduces scatter in retrieved CO2 rather than focusing on the more complex (and easily removed) bias issues.

Mandrake, Lukas↗

Global Sensitivity Analysis of Simulated Remote Sensing Polarimetric Observations Over Snow

This study presents a detailed theoretical assessment of the information content of passive polarimetric observations over snow scenes, using a global sensitivity analysis (GSA) method. Conventional sensitivity studies focus on varying a single parameter while keeping all other parameters fixed. In contrast, the GSA correctly addresses the covariance of state parameters across their entire parameter space, hence favoring a more correct interpretation of inversion algorithms and the optimal design of their state vectors. The forward simulations exploit a vector radiative transfer model to obtain the Stokes vector emerging at the top of the atmosphere for different solar zenith angles, when the bottom boundary consists of a vertically resolved snowpack of non-spherical grains. The presence of light-absorbing particulates (LAPs), either embedded in the snow or aloft in the atmosphere above in the form of aerosols, is also considered. The results are presented for a set of wavelengths spanning the visible (VIS), near-infrared (NIR), and shortwave infrared (SWIR) region of the spectrum. The GSA correctly captures the expected, high sensitivity of the reflectance to LAPs in the VIS–NIR and to grain size at different depths in the snowpack in the NIR–SWIR. With adequate viewing geometries, mono-angle measurements of total reflectance in the VIS–SWIR (akin to those of the Moderate Resolution Imaging Spectroradiometer, MODIS) resolve grain size in the top layer of the snowpack sufficiently well. The addition of multi-angle polarimetric observations in the VIS–NIR provides information on grain shape and microscale roughness. The simultaneous sensitivity in the VIS–NIR to both aerosols and snow-embedded impurities can be disentangled by extending the spectral range to the SWIR, which contains information on aerosol optical depth while remaining essentially unaffected when the same particulates are mixed with the snow. Multi-angle polarimetric observations can therefore (i) effectively partition LAPs between the atmosphere and the surface, which represents a notorious challenge for snow remote sensing based on measurements of total reflectance only and (ii) lead to better estimates of grain shape and roughness and, in turn, the asymmetry parameter, which is critical for the determination of albedo. The retrieval uncertainties are minimized when the degree of linear polarization is used in place of the polarized reflectance. The Sobol indices, which are the main metric for the GSA, were used to select the state parameters in retrievals performed on data simulated for multiple instrument configurations. Improvements in retrieval quality with the addition of measurements of polarization, multi-angle views, and different spectral channels reflect the information content, identified by the Sobol indices, relative to each configuration. The results encourage the development of new remote sensing algorithms that fully leverage multi-angle and polarimetric capabilities of modern remote sensors. They can also aid flight planning activities, since the optimal exploitation of the information content of multi-angle measurements depends on the viewing geometry. The better characterization of surface and atmospheric parameters in snow-covered regions advances research opportunities for scientists of the cryosphere and ultimately benefits albedo estimates in climate models.

remote sensing↗

Tonlé Sap Food Security & Agriculture III: Evaluating Changes in Ecosystem Vitality and Freshwater Health in the Tonlé Sap Basin using Remotely Sensed Data

Tonlé Sap Lake, located in Cambodia, Southeast Asia, is one of the most productive inland fisheries in the world. With the unique reverse flow hydrology of the Tonlé Sap River, this freshwater system contains rich biodiversity and provides critical freshwater resources for the local community. Overfishing, stronger seasonality, drought, dam construction, forest fires, and untreated industrial domestic sewage threaten the ecosystem vitality and economic success of Tonlé Sap. In collaboration with Conservation International and the Cambodian Ministry of Water Resources and Meteorology’s Tonlé Sap Authority, we calibrated and finalized remotely-sensed proxies for sub-indicators of the Freshwater Health Index (FHI). We used NASA Earth observation data from Moderate Resolution Imaging Spectroradiometer (MODIS), and the Gravity Recovery and Climate Experiment (GRACE). These datasets were used in RS proxies and a Soil and Water Assessment Tool (SWAT) model that previous teams developed for sub-indicators of FHI. They included landcover, bank modification, and water quality metrics. It was determined that the ground water storage in Tonlé Sap Lake exhibited a slight declining trend over from April 1st, 2002 to February 3rd, 2017. To calculate chlorophyll-a concentration in the Tonlé Sap Lake, we used Copernicus Global Land Service mission data which provided a Trophic State Index of the lake. Between January 1st, 2000 and December 31st, 2020, the lake and surrounding region within the study area boundary displayed a slight decrease in vegetation density and consistently high chlorophyll-a concentrations. The SWAT model calculated nitrogen and phosphorus content measured in outlet points of the lake. We demonstrated that remotely sensed data is valuable for providing additional information for the FHI, but is not fully capable of replacing its in situ counterpart. We also concluded the water quantity is on a slowly declining trend within the basin.

Joey Lindsay↗

Using Quality Attributes to Bridge Systems Engineering Gaps : A Juno Ground Data Systems Case Study

The Juno Mission to Jupiter is the second mission selected by the NASA New Frontiers Program. Juno launched August 2011 and will reach Jupiter July 2016. Juno's payload system is composed of nine instruments plus a gravity science experiment. One of the primary functions of the Juno Ground Data System (GDS) is the assembly and distribution of the CFDP (CCSDS File Delivery Protocol) product telemetry, also referred to as raw science data, for eight out of the nine instruments. The GDS accomplishes this with the Instrument Data Pipeline (IDP). During payload integration, the first attempt to exercise the IDP in a flight like manner revealed that although the functional requirements were well understood, the system was unable to meet latency requirements with the as-is heritage design. A systems engineering gap emerged between Juno instrument data delivery requirements and the assumptions behind the heritage flight-ground interactions. This paper describes the use of quality attributes to measure and overcome this gap by introducing a new systems engineering activity, and a new monitoring service architecture that successfully delivered the performance metrics needed to validate Juno IDP.

Ground Data Systems (GDS)↗

Validating LES for Jet Aeroacoustics

Engineers charged with making jet aircraft quieter have long dreamed of being able to see exactly how turbulent eddies produce sound and this dream is now coming true with the advent of large eddy simulation (LES). Two obvious challenges remain: validating the LES codes at the resolution required to see the fluid-acoustic coupling, and the interpretation of the massive datasets that are produced. This paper addresses the former, the use of advanced experimental techniques such as particle image velocimetry (PIV) and Raman and Rayleigh scattering, to validate the computer codes and procedures used to create LES solutions. This paper argues that the issue of accuracy of the experimental measurements be addressed by cross-facility and cross-disciplinary examination of modern datasets along with increased reporting of internal quality checks in PIV analysis. Further, it argues that the appropriate validation metrics for aeroacoustic applications are increasingly complicated statistics that have been shown in aeroacoustic theory to be critical to flow-generated sound, such as two-point space-time velocity correlations. A brief review of data sources available is presented along with examples illustrating cross-facility and internal quality checks required of the data before it should be accepted for validation of LES.

Bridges, James↗

Long-Term Evaluation of Landsat 9 Oli Data Product Uniformity at Focal Plane Module-Boundaries Using Earth Scenes

The uniformity performance characteristic in remote sensing data products is a metric that helps evaluate how an image accurately represents radiometric responses to geophysical variability at the Earth’s surface or within the atmosphere. These uniformity attributes are used in Earth remote sensing science applications such as investigating water quality, land-cover/land-use classifications, and crop health assessments. To ensure these applications perform effectively, it is crucial to maintain and regularly assess the uniformity of calibrated remote sensing data products. In the Landsat missions, data from the onboard radiometric reference Solar diffuser device are used as the standard approach to derive the calibration parameters, ensuring the maintenance of both absolute radiance and radiometric uniformity through quarterly updates. This publication presents an alternative approach that leverages Earth scene data to assess the on-orbit uniformity performance of the Landsat 9 (L9) mission. The analysis characterizes uniformity at a specific set of spatial positions (the focal-plane boundary zones) that are intrinsic to the Operational Land Imager (OLI) focal plane design. By employing these Earth data statistics, the article further explores the temporal trends in the radiometric uniformity results. The results shown in this article demonstrate that for pushbroom imaging design systems with staggered sensor chip focal plane architectures, the co-registered pixels, i.e., overlapping measurements of the same ground target viewed by different detectors, offer a self-contained mechanism for assessing product uniformity. Remarkably, this alternative approach achieves a level of radiometric quality comparable to that delivered by a stable, well-characterized onboard Solar diffuser calibration device. The findings demonstrate the potential and capability of such Earth scene statistics in maintaining radiometric uniformity in calibrated data products to better than 1% (1-sigma) throughout the mission’s operational lifetime.

trend↗

Force Measurement Improvements to the National Transonic Facility Sidewall Model Support System

The National Transonic Facility is a transonic pressurized cryogenic facility. The development of the high Reynolds number semi-span capability has advanced over the years to include transonic active flow control and powered testing using the sidewall model support system. While this system can be used in total temperatures down to -250Â F for conventional unpowered configurations, it is limited to temperatures above -60Â F when used with powered models that require the use of the high-pressure air delivery system. Thermal instabilities and non-repeatable mechanical arrangements revealed several data quality shortfalls by the force and moment measurement system. Recent modifications to the balance cavity recirculation system have improved the temperature stability of the balance and metric model-to-balance hardware. Changes to the mechanical assembly of the high-pressure air delivery system, particularly hardware that interfaces directly with the model and balance, have improved the repeatability of the force and moment measurement system. Drag comparisons with the high-pressure air system removed will also be presented in this paper.

Goodliff, Scott L.↗

Desert Research and Technology Studies (DRATS) 2010 Science Operations: Operational Approaches and Lessons Learned for Managing Science during Human Planetary Surface Missions

Desert Research and Technology Studies (Desert RATS) is a multi-year series of hardware and operations tests carried out annually in the high desert of Arizona on the San Francisco Volcanic Field. These activities are designed to exercise planetary surface hardware and operations in conditions where long-distance, multi-day roving is achievable, and they allow NASA to evaluate different mission concepts and approaches in an environment less costly and more forgiving than space.The results from the RATS tests allows election of potential operational approaches to planetary surface exploration prior to making commitments to specific flight and mission hardware development. In previous RATS operations, the Science Support Room has operated largely in an advisory role, an approach that was driven by the need to provide a loose science mission framework that would underpin the engineering tests. However, the extensive nature of the traverse operations for 2010 expanded the role of the science operations and tested specific operational approaches. Science mission operations approaches from the Apollo and Mars-Phoenix missions were merged to become the baseline for this test. Six days of traverse operations were conducted during each week of the 2-week test, with three traverse days each week conducted with voice and data communications continuously available, and three traverse days conducted with only two 1-hour communications periods per day. Within this framework, the team evaluated integrated science operations management using real-time, tactical science operations to oversee daily crew activities, and strategic level evaluations of science data and daily traverse results during a post-traverse planning shift. During continuous communications, both tactical and strategic teams were employed. On days when communications were reduced to only two communications periods per day, only a strategic team was employed. The Science Operations Team found that, if communications are good and down-linking of science data is ensured, high quality science returns is possible regardless of communications. What is absent from reduced communications is the scientific interaction between the crew on the planet and the scientists on the ground. These scientific interactions were a critical part of the science process and significantly improved mission science return over reduced communications conditions. The test also showed that the quality of science return is not measurable by simple numerical quantities but is, in fact, based on strongly non-quantifiable factors, such as the interactions between the crew and the Science Operations Teams. Although the metric evaluation data suggested some trends, there was not sufficient granularity in the data or specificity in the metrics to allow those trends to be understood on numerical data alone.

Eppler, Dean↗

Quality Assessment of Landsat Surface Reflectance Products Using MODIS Data

Surface reflectance adjusted for atmospheric effects is a primary input for land cover change detection and for developing many higher level surface geophysical parameters. With the development of automated atmospheric correction algorithms, it is now feasible to produce large quantities of surface reflectance products using Landsat images. Validation of these products requires in situ measurements, which either do not exist or are difficult to obtain for most Landsat images. The surface reflectance products derived using data acquired by the Moderate Resolution Imaging Spectroradiometer (MODIS), however, have been validated more comprehensively. Because the MODIS on the Terra platform and the Landsat 7 are only half an hour apart following the same orbit, and each of the 6 Landsat spectral bands overlaps with a MODIS band, good agreements between MODIS and Landsat surface reflectance values can be considered indicators of the reliability of the Landsat products, while disagreements may suggest potential quality problems that need to be further investigated. Here we develop a system called Landsat-MODIS Consistency Checking System (LMCCS). This system automatically matches Landsat data with MODIS observations acquired on the same date over the same locations and uses them to calculate a set of agreement metrics. To maximize its portability, Java and open-source libraries were used in developing this system, and object-oriented programming (OOP) principles were followed to make it more flexible for future expansion. As a highly automated system designed to run as a stand-alone package or as a component of other Landsat data processing systems, this system can be used to assess the quality of essentially every Landsat surface reflectance image where spatially and temporally matching MODIS data are available. The effectiveness of this system was demonstrated using it to assess preliminary surface reflectance products derived using the Global Land Survey (GLS) Landsat images for the 2000 epoch. As surface reflectance likely will be a standard product for future Landsat missions, the approach developed in this study can be adapted as an operational quality assessment system for those missions.

Landsat↗

Morphometric Characterization of Lunar Landing Sites

As ambitious surface exploration of the Moon commences in the 2020s, it is important to develop reliable and objective metrics for understanding the quality of future landing sites. A key prerequisite for any exploration and utilization of the lunar surface is a safe landing. One of the most important methods to provide understanding of potential metrics for landing site safety in the lunar context is the systematic comparison of candidate and historic landing sites. The goal of this project is to determine the morphometric parameters of the lunar surface at 16 successful lunar landing sites where adequate data exists to execute a quantitative comparison using various parameters. Quantitative analysis of Lunar Reconnaissance Orbiter (LRO) data should inform mission planning activities by providing morphologic metrics for landing sites including slope, Terrain Ruggedness Index (TRI), and rock abundance. These comparisons will assist mission planners and exploration scientists by providing “calibration points” for using Lunar Reconnaissance Orbiter (LRO) data to successfully plan and execute lunar powered descents in the future. The metrics included in this analysis describe the form of the terrain and can be calculated for any potential future landing site. Another objective of this project is to determine how the derived morphologic parameters for the same landing site change between common pixel scales. In ideal lighting conditions, 2m/px Narrow Angle Camera Digital Terrain Models (NAC DTMs) can be assembled using stereoscopic imagery from two or more concentric orbits of the LRO. However, due to low light conditions near the lunar poles, the highest quality data for many potential lunar landing sites comes from the Lunar Orbiter Laser Altimeter (LOLA). Therefore, we seek to assess how the derived slope and TRI values change as a function of changes in the DTM postings from 2m/px to 5m/px.

J M McCallion↗

Spatiotemporal Coherence of Mean and Extreme Aerosol Particle Events over Eastern North America as Observed from Satellite

Quantifying spatiotemporal scales of coherence and variability of aerosol particle properties provides (i) metrics for evaluating observational data sets and model simulations, (ii) inferences regarding dominant causes of variability, and (iii) information that is necessary to determine their climate forcing and air quality impacts. Accordingly, analyses of columnar aerosol optical depth (AOD) and Ångstrom exponent (AE) from the MODerate resolution Imaging Spectroradiometer (MODIS) on the Aqua and Terra satellites, and AErosol RObotic NETwork (AERONET) stations are used to evaluate the spatially averaged (Level-3) MODIS products, and to characterize the spatiotemporal scales of variability (and coherence) of aerosol particles over eastern North America (2000-2013). Results show a high degree of consistency in AOD retrievals from the different data sets but larger discrepancies in AE estimates, highest mean AOD and lowest day-to-day variability during the summer, and largest scales of spatial coherence in summer and fall. Consistent with the scales of spatial coherence from MODIS data, in both data sets power spectra indicate AOD variability is manifest primarily on synoptic and annual time scales. Conversely, AE variability in MODIS data is primarily focused on seasonal, semiannual, and annual time scales, and there is an additional mode of AE variability at approx. 30 days in the AERONET measurements. The frequency of cooccurrence of extreme AOD values (>local 90th percentile) decreases to below 50% at approx. 150 km from a central grid cell, but is above that expected by random chance over almost all of eastern North America, indicating supra-regional scale extreme events.

MODIS↗

Teamwork for Oversight of Processes and Systems (TOPS). Implementation guide for TOPS version 2.0, 10 August 1992

As the nation redefines priorities to deal with a rapidly changing world order, both government and industry require new approaches for oversight of management systems, particularly for high technology products. Declining defense budgets will lead to significant reductions in government contract management personnel. Concurrently, defense contractors are reducing administrative and overhead staffing to control costs. These combined pressures require bold approaches for the oversight of management systems. In the Spring of 1991, the DPRO and TRW created a Process Action Team (PAT) to jointly prepare a Performance Based Management (PBM) system titled Teamwork for Oversight of Processes and Systems (TOPS). The primary goal is implementation of a performance based management system based on objective data to review critical TRW processes with an emphasis on continuous improvement. The processes are: Finance and Business Systems, Engineering and Manufacturing Systems, Quality Assurance, and Software Systems. The team established a number of goals: delivery of quality products to contractual terms and conditions; ensure that TRW management systems meet government guidance and good business practices; use of objective data to measure critical processes; elimination of wasteful/duplicative reviews and audits; emphasis on teamwork--all efforts must be perceived to add value by both sides and decisions are made by consensus; and synergy and the creation of a strong working trust between TRW and the DPRO. TOPS permits the adjustment of oversight resources when conditions change or when TRW systems performance indicate either an increase or decrease in surveillance is appropriate. Monthly Contractor Performance Assessments (CPA) are derived from a summary of supporting system level and process-level ratings obtained from objective process-level data. Tiered, objective, data-driven metrics are highly successful in achieving a cooperative and effective method of measuring performance. The teamwork-based culture developed by TOPS proved an unequaled success in removing adversarial relationships and creating an atmosphere of continuous improvement in quality processes at TRW. The new working relationship does not decrease the responsibility or authority of the DPRO to ensure contract compliance and it permits both parties to work more effectively to improve total quality and reduce cost. By emphasizing teamwork in developing a stronger approach to efficient management of the defense industrial base TOPS is a singular success.

Strand, Albert A.↗