Search NASA⌕ Search

SEARCH · Search NASA

Results for “validations”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Systematic Independent Validation of Inner Heliospheric Models

This presentation is the first in a series which will provide independent validation of community models of the outer corona and inner heliosphere. In this work we establish a set of measures to be used in validating this group of models. We use these procedures to generate a comprehensive set of results from the Wang- Sheeley-Arge (WSA) model which will be used as a baseline, or reference, against which to compare all other models. We also run a test of the validation procedures by applying them to a small set of results produced by the ENLIL Magnetohydrodynamic (MHD) model. In future presentations we will validate other models currently hosted by the Community Coordinated Modeling Center(CCMC), including a comprehensive validation of the ENLIL model. The Wang-Sheeley-Arge (WSA) model is widely used to model the Solar wind, and is used by a number of agencies to predict Solar wind conditions at Earth as much as four days into the future. Because it is so important to both the research and space weather forecasting communities, it is essential that its performance be measured systematically, and independently. In this paper we offer just such an independent and systematic validation. We report skill scores for the model's predictions of wind speed and IMF polarity for a large set of Carrington rotations. The model was run in all its routinely used configurations. It ingests line of sight magnetograms. For this study we generated model results for monthly magnetograms from the National Solar Observatory (SOLIS), Mount Wilson Observatory and the GONG network, spanning the Carrington rotation range from 1650 to 2068. We compare the influence of the different magnetogram sources, performance at quiet and active times, and estimate the effect of different empirical wind speed tunings. We also consider the ability of the WSA model to identify sharp transitions in wind speed from slow to fast wind. These results will serve as a baseline against which to compare future versions of the model

MacNeice, P. J.↗

Definition and Demonstration of a Methodology for Validating Aircraft Trajectory Predictors

This paper presents a new methodology for validating an aircraft trajectory predictor, inspired by the lessons learned from a number of field trials, flight tests and simulation experiments for the development of trajectory-predictor-based automation. The methodology introduces new techniques and a new multi-staged approach to reduce the effort in identifying and resolving validation failures, avoiding the potentially large costs associated with failures during a single-stage, pass/fail approach. As a case study, the validation effort performed by the Federal Aviation Administration for its En Route Automation Modernization (ERAM) system is analyzed to illustrate the real-world applicability of this methodology. During this validation effort, ERAM initially failed to achieve six of its eight requirements associated with trajectory prediction and conflict probe. The ERAM validation issues have since been addressed, but to illustrate how the methodology could have benefited the FAA effort, additional techniques are presented that could have been used to resolve some of these issues. Using data from the ERAM validation effort, it is demonstrated that these new techniques could have identified trajectory prediction error sources that contributed to several of the unmet ERAM requirements.

Vivona, Robert A.↗

Turbulence Modeling Verification and Validation

Computational fluid dynamics (CFD) software that solves the Reynolds-averaged Navier-Stokes (RANS) equations has been in routine use for more than a quarter of a century. It is currently employed not only for basic research in fluid dynamics, but also for the analysis and design processes in many industries worldwide, including aerospace, automotive, power generation, chemical manufacturing, polymer processing, and petroleum exploration. A key feature of RANS CFD is the turbulence model. Because the RANS equations are unclosed, a model is necessary to describe the effects of the turbulence on the mean flow, through the Reynolds stress terms. The turbulence model is one of the largest sources of uncertainty in RANS CFD, and most models are known to be flawed in one way or another. Alternative methods such as direct numerical simulations (DNS) and large eddy simulations (LES) rely less on modeling and hence include more physics than RANS. In DNS all turbulent scales are resolved, and in LES the large scales are resolved and the effects of the smallest turbulence scales are modeled. However, both DNS and LES are too expensive for most routine industrial usage on today's computers. Hybrid RANS-LES, which blends RANS near walls with LES away from walls, helps to moderate the cost while still retaining some of the scale-resolving capability of LES, but for some applications it can still be too expensive. Even considering its associated uncertainties, RANS turbulence modeling has proved to be very useful for a wide variety of applications. For example, in the aerospace field, many RANS models are considered to be reliable for computing attached flows. However, existing turbulence models are known to be inaccurate for many flows involving separation. Research has been ongoing for decades in an attempt to improve turbulence models for separated and other nonequilibrium flows. When developing or improving turbulence models, both verification and validation are important steps in the process. Verification insures that the CFD code is solving the equations as intended (no errors in the implementation). This is typically done either through the method of manufactured solutions (MMS) or through careful step-by-step comparisons with other verified codes. After the verification step is concluded, validation is performed to document the ability of the turbulence model to represent different types of flow physics. Validation can involve a large number of test case comparisons with experiments, theory, or DNS. Organized workshops have proved to be valuable resources for the turbulence modeling community in its pursuit of turbulence modeling verification and validation. Workshop contributors using different CFD codes run the same cases, often according to strict guidelines, and compare results. Through these comparisons, it is often possible to (1) identify codes that have likely implementation errors, and (2) gain insight into the capabilities and shortcomings of different turbulence models to predict the flow physics associated with particular types of flows. These are valuable lessons because they help bring consistency to CFD codes by encouraging the correction of faulty programming and facilitating the adoption of better models. They also sometimes point to specific areas needed for improvement in the models. In this paper, several recent workshops are summarized primarily from the point of view of turbulence modeling verification and validation. Furthermore, the NASA Langley Turbulence Modeling Resource website is described. The purpose of this site is to provide a central location where RANS turbulence models are documented, and test cases, grids, and data are provided. The goal of this paper is to provide an abbreviated survey of turbulence modeling verification and validation efforts, summarize some of the outcomes, and give some ideas for future endeavors in this area.

Rumsey, Christopher L.↗

System-Level Experimental Validations for Supersonic Commercial Transport Aircraft Entering Service in the 2018-2020 Time Period

This report describes the work conducted under NASA funding for the Boeing N+2 Supersonic Experimental Validation project to experimentally validate the conceptual design of a supersonic airliner feasible for entry into service in the 2018 -to 2020 timeframe (NASA N+2 generation). The primary goal of the project was to develop a low-boom configuration optimized for minimum sonic boom signature (65 to 70 PLdB). This was a very aggressive goal that could be achieved only through integrated multidisciplinary optimization tools validated in relevant ground and, later, flight environments. The project was split into two phases. Phase I of the project covered the detailed aerodynamic design of a low boom airliner as well as the wind tunnel tests to validate that design (ref. 1). This report covers Phase II of the project, which continued the design methodology development of Phase I with a focus on the propulsion integration aspects as well as the testing involved to validate those designs. One of the major airplane configuration features of the Boeing N+2 low boom design was the overwing nacelle. The location of the nacelle allowed for a minimal effect on the boom signature, however, it added a level of difficulty to designing an inlet with acceptable performance in the overwing flow field. Using the Phase I work as the starting point, the goals of the Phase 2 project were to design and verify inlet performance while maintaining a low-boom signature. The Phase II project was successful in meeting all contract objectives. New modular nacelles were built for the larger Performance Model along with a propulsion rig with an electrically-actuated mass flow plug. Two new mounting struts were built for the smaller Boom Model, along with new nacelles. Propulsion integration testing was performed using an instrumented fan face and a mass flow plug, while boom signatures were measured using a wall-mounted pressure rail. A side study of testing in different wind tunnels was completed as a precursor to the selection of the facilities used for validation testing. As facility schedules allowed, the propulsion testing was done at the NASA Glenn Research Center (GRC) 8 x 6-Foot wind tunnel, while boom and force testing was done at the NASA Ames Research Center (ARC) 9 x 7-Foot wind tunnel. During boom testing, a live balance was used for gathering force data. This report is broken down into nine sections. The first technical section (Section 2) covers the general scope of the Phase II activities, goals, a description of the design and testing efforts, and the project plan and schedule. Section 3 covers the details of the propulsion system concepts and design evolution. A series of short tests to evaluate the suitability of different wind tunnels for boom, propulsion, and force testing was also performed under the Phase 2 effort, with the results covered in Section 4. The propulsion integration testing is covered in Section 5 and the boom and force testing in Section 6. CFD comparisons and analyses are included in Section 7. Section 8 includes the conclusions and lessons learned.

Magee, Todd E.↗

Soil Moisture Active Passive Mission L4_C Data Product Assessment (Version 2 Validated Release)

The SMAP satellite was successfully launched January 31st 2015, and began acquiring Earth observation data following in-orbit sensor calibration. Global data products derived from the SMAP L-band microwave measurements include Level 1 calibrated and geolocated radiometric brightness temperatures, Level 23 surface soil moisture and freezethaw geophysical retrievals mapped to a fixed Earth grid, and model enhanced Level 4 data products for surface to root zone soil moisture and terrestrial carbon (CO2) fluxes. The post-launch SMAP mission CalVal Phase had two primary objectives for each science product team: 1) calibrate, verify, and improve the performance of the science algorithms, and 2) validate accuracies of the science data products as specified in the L1 science requirements. This report provides analysis and assessment of the SMAP Level 4 Carbon (L4_C) product pertaining to the validated release. The L4_C validated product release effectively replaces an earlier L4_C beta-product release (Kimball et al. 2015). The validated release described in this report incorporates a longer data record and benefits from algorithm and CalVal refinements acquired during the SMAP post-launch CalVal intensive period. The SMAP L4_C algorithms utilize a terrestrial carbon flux model informed by SMAP soil moisture inputs along with optical remote sensing (e.g. MODIS) vegetation indices and other ancillary biophysical data to estimate global daily net ecosystem CO2 exchange (NEE) and component carbon fluxes for vegetation gross primary production (GPP) and ecosystem respiration (Reco). Other L4_C product elements include surface (10 cm depth) soil organic carbon (SOC) stocks and associated environmental constraints to these processes, including soil moisture and landscape freeze/thaw (FT) controls on GPP and respiration (Kimball et al. 2012). The L4_C product encapsulates SMAP carbon cycle science objectives by: 1) providing a direct link between terrestrial carbon fluxes and underlying FT and soil moisture constraints to these processes, 2) documenting primary connections between terrestrial water, energy and carbon cycles, and 3) improving understanding of terrestrial carbon sink activity in northern ecosystems. There are no L1 science requirements for the L4_C product; however self-imposed requirements have been established focusing on NEE as the primary product field for validation, and on demonstrating L4_C accuracy and success in meeting product science requirements (Jackson et al. 2012). The other L4_C product fields also have strong utility for carbon science applications; however, analysis of these other fields is considered secondary relative to primary validation activities focusing on NEE. The L4_C targeted accuracy requirements are to meet or exceed a mean unbiased accuracy (ubRMSE) for NEE of 1.6 g C/sq m/d or 30 g C/sq m/yr, emphasizing northern (45N) boreal and arctic ecosystems; this is similar to the estimated accuracy level of in situ tower eddy covariance measurement-based observations (Baldocchi 2008).

Cal/Val↗

Validation of MODIS 3 km Land Aerosol Optical Depth from NASA's EOS Terra and Aqua Missions

In addition to the standard resolution product (10km), the MODerate resolution Imaging Spectroradiometer (MODIS) Collection 6 (C006) data release included a higher resolution (3km). Other than accommodations for the two different resolutions, the 10 and 3km Dark Target (DT) algorithms are basically the same. In this study, we perform global validation of the higher-resolution aerosol optical depth (AOD) over global land by comparing against AErosol RObotic NETwork (AERONET) measurements. The MODIS-AERONET collocated data sets consist of 161410 high-confidence AOD pairs from 2000 to 2015 for Terra MODIS and 2003 to 2015 for Aqua MODIS. We find that 62.5 and 68.4% of AODs retrieved from Terra MODIS and Aqua MODIS, respectively, fall within previously published expected error bounds of +/-(0.05+0.2 × AOD), with a high correlation (R = 0.87). The scatter is not random, but exhibits a mean positive bias of ∼ 0.06 for Terra and ∼ 0.03 for Aqua. These biases for the 3km product are approximately 0.03 larger than the biases found in similar validations of the 10km product. The validation results for the 3km product did not have a relationship to aerosol loading (i.e., true AOD), but did exhibit dependence on quality flags, region, viewing geometry, and aerosol spatial variability. Time series of global MODIS-AERONET differences show that validation is not static, but has changed over the course of both sensors' lifetimes, with Terra MODIS showing more change over time. The likely cause of the change of validation over time is sensor degradation, but changes in the distribution of AERONET stations and differences in the global aerosol system itself could be contributing to the temporal variability of validation.

Gupta, Pawan↗

System Validation on the Europa Clipper mission in Early Implementation Phase

NASA’s next flag-ship mission - Europa Clipper, will embark on a journey to Jupiter’s icy moon Europa in 2024 to assess its environment and habitability with a highly capable spacecraft. Post Jupiter-Orbit-Insertion, the spacecraft will be commanded to perform intricate, yet meticulously planned Europa flybys to perform science investigations using a suite of instruments, while withstanding Jupiter’s harsh radiation environment. The success of this mission is dependent on a well-coordinated project and its elements such as the flight hardware and software, the ground support and mission operations teams, procedures and other cross-cutting elements. The Europa Clipper project needs to ensure that these elements are realized at a reasonable confidence level prior to launch and other mission critical events. System Validation test and analysis activities exercise and confirm the integrity of the system of all project elements in the expected flight environment with reasonable stressing conditions. These activities go beyond system design requirements verification and are driven by validation objectives that describe the end-to-end functional and operational capabilities required during nominal and off-nominal flight-like scenarios and critical events. The challenges associated with validating that the Europa Clipper project as a whole can function and perform correctly to meet the intended mission objectives with the as-delivered capabilities of all of its elements are daunting. This paper discusses the systematic methodology established in the early implementation phase of the Europa Clipper project for developing System Validation activities and their validation objectives, and addressing any validation-related challenges on the project. Approaches include decomposition of mission objectives using activity timelines in the Mission Design plan for developing nominal scenarios, use of fault trees for exploring off-nominal cases and system boundaries, and use of Model-based Systems Engineering (MBSE) tools for planning and prioritizing these activities.

Wang, Xu↗

Spectral and Spatial Dependencies in the Validation of Satellite-Based Aerosol Optical Depth from the Geostationary Ocean Color Imager Using the Aerosol Robotic Network

The regional and global scale of aerosols in the atmosphere can be quantified using the aerosol optical depth (AOD) retrieved from satellite observations. To obtain reliable satellite AODs, conducting consistent validations and refining retrieval algorithms are crucial. AODs and Ångström exponents (AEs) measured with the aerosol robotic network (AERONET) are considered as the ground truth for satellite validations. AERONET AEs are used to collocate the wavelength of the AERONET AODs to those of the satellite AODs when there is a discordancy in their wavelengths. However, numerous validation studies have proposed different strategies by applying the AERONET AODs and AEs, and spatiotemporal collocation criteria. This study examined the impact of the wavelength and spatial collocation radius variations by comparing AODs at 550 nm derived from the geostationary ocean color imager (GOCI) with those obtained from the AERONET for the year 2016. The estimated AERONET AODs at 550 nm varied from 5.18% to 11.73% depending on the selection of AOD and AE, and the spatial collocation radii from 0 to 40 km, respectively. The longer the collocation radius and the higher the AODs, the greater the variability observed in the validation results. Overall, the selection of the spatial collocation radius had a stronger impact on the variability in the validation results obtained compared to the selection of the wavelength. The variability was also found in seasonal analysis. Therefore, it is recommended to carefully select the data wavelength and spatial collocation radius, consider seasonal effects, and provide this information when validating satellite AODs using AERONET.

GOCI↗

Validation (not just verification) of Deep Space Missions

ion & Validation (V&V) is a widely recognized and critical systems engineering function. However, the often used definition 'Verification proves the design is right; validation proves it is the right design' is rather vague. And while Verification is a reasonably well standardized systems engineering process, Validation is a far more abstract concept and the rigor and scope applied to it varies widely between organizations and individuals. This is reflected in the findings in recent Mishap Reports for several NASA missions, in which shortfalls in Validation (not just Verification) were cited as root- or contributing-factors in catastrophic mission loss. Furthermore, although there is strong agreement in the community that Test is the preferred method for V&V, many people equate 'V&V' with 'Test', such that Analysis and Modeling aren't given comparable attention. Another strong motivator is a realization that the rapid growth in complexity of deep-space missions (particularly Planetary Landers and Space Observatories given their inherent unknowns) is placing greater demands on systems engineers to 'get it right' with Validation.

verification↗

Validation Database Based Thermal Analysis of an Advanced RPS Concept

Advanced RPS concepts can be conceived, designed and assessed using high-end computational analysis tools. These predictions may provide an initial insight into the potential performance of these models, but verification and validation are necessary and required steps to gain confidence in the numerical analysis results. This paper discusses the findings from a numerical validation exercise for a small advanced RPS concept, based on a thermal analysis methodology developed at JPL and on a validation database obtained from experiments performed at Oregon State University. Both the numerical and experimental configurations utilized a single GPHS module enabled design, resembling a Mod-RTG concept. The analysis focused on operating and environmental conditions during the storage phase only. This validation exercise helped to refine key thermal analysis and modeling parameters, such as heat transfer coefficients, and conductivity and radiation heat transfer values. Improved understanding of the Mod-RTG concept through validation of the thermal model allows for future improvements to this power system concept.

numerical model↗

In Situ Validation of the Soil Moisture Active Passive (SMAP) Satellite Mission

SMAP is a new NASA mission proposed for 2014 that would provide a number of soil moisture and freeze/thaw products. The soil moisture products span spatial resolutions from 3 to 40 km. In situ soil moisture observations will be one of the key elements of the validation program for SMAP. Data from the currently available set of soil moisture observing sites and networks need improvement if they are to be useful. Problems include a lack of standardization of instrumentation and installation and the disparity in spatial scale between the point-scale in situ data (a few centimeters) and the coarser satellite products. SMAP has initiated activities to resolve these issues for some of the existing resources. The other challenge to soil moisture validation is the need to expand the number of sites and their geographic distribution. SMAP is attempting to increase the number of sites and their value in validation through collaboration. The issues and solutions involving in situ validation being investigated will be described along with recent results from SMAP validation projects.

validation↗

Verification and Validation Studies for the LAVA CFD Solver

The verification and validation of the Launch Ascent and Vehicle Aerodynamics (LAVA) computational fluid dynamics (CFD) solver is presented. A modern strategy for verification and validation is described incorporating verification tests, validation benchmarks, continuous integration and version control methods for automated testing in a collaborative development environment. The purpose of the approach is to integrate the verification and validation process into the development of the solver and improve productivity. This paper uses the Method of Manufactured Solutions (MMS) for the verification of 2D Euler equations, 3D Navier-Stokes equations as well as turbulence models. A method for systematic refinement of unstructured grids is also presented. Verification using inviscid vortex propagation and flow over a flat plate is highlighted. Simulation results using laminar and turbulent flow past a NACA 0012 airfoil and ONERA M6 wing are validated against experimental and numerical data.

Validation↗

Validation of IASI Satellite Ammonia Observations at the Pixel Scale Using In Situ Vertical Profiles

Satellite ammonia (NH 3 ) observations provide unprecedented insights into NH3 emissions, spatiotemporal variabilities and trends, but validation with in situ measurements remains lacking. Here, total columns from the Infrared Atmospheric Sounding Interferometer (IASI) were intercompared to boundary layer NH 3 profiles derived from aircraft- and surface-based measurements primarily in Colorado, USA, in the summer of 2014. IASI-NH 3 version 3 near real-time data set compared well to in situ derived columns (windows ±15 km around centroid, ±1 h around overpass time) with a correlation of 0.58, a slope of 0.78 ± 0.14 and an intercept of 2.1 × 10 15 ±1.5 × 10 15 molecules cm −2 . Agreement degrades at larger spatiotemporal windows, consistent with the short atmospheric lifetime of NH 3 . We also examined IASI version 3R data, which relies on temperature retrievals from the ERA Reanalysis, and a third product generated using aircraft-measured temperature profiles. The overall agreement improves slightly for both cases, and neither is biased within their combined measurement errors. Thus, spatiotemporal averaging of IASI over large windows can be used to reduce retrieval noise. Nonetheless, sampling artifacts of airborne NH 3 instruments result in significant uncertainties of the in situ-derived columns. For example, large validation differences exist between ascent and descent profiles, and the assumptions of the free tropospheric NH 3 profiles used above the aircraft ceiling significantly impact the validation. Because short-lived species like NH 3 largely reside within the boundary layer with complex vertical structures, more comprehensive validation is needed across a wide range of environments. More accurate and widespread in situ NH 3 data sets are therefore required for improved validations of satellite products.

Ammonia↗

Validation Methods Research for Fault-Tolerant Avionics and Control Systems: Working Group Meeting, 2

The validation process comprises the activities required to insure the agreement of system realization with system specification. A preliminary validation methodology for fault tolerant systems documented. A general framework for a validation methodology is presented along with a set of specific tasks intended for the validation of two specimen system, SIFT and FTMP. Two major areas of research are identified. First, are those activities required to support the ongoing development of the validation process itself, and second, are those activities required to support the design, development, and understanding of fault tolerant systems.

Gault, J. W.↗

A validation methodology for fault-tolerant clock synchronization

A validation method for the synchronization subsystem of a fault-tolerant computer system is presented. The high reliability requirement of flight crucial systems precludes the use of most traditional validation methods. The method presented utilizes formal design proof to uncover design and coding errors and experimentation to validate the assumptions of the design proof. The experimental method is described and illustrated by validating an experimental implementation of the Software Implemented Fault Tolerance (SIFT) clock synchronization algorithm. The design proof of the algorithm defines the maximum skew between any two nonfaulty clocks in the system in terms of theoretical upper bounds on certain system parameters. The quantile to which each parameter must be estimated is determined by a combinatorial analysis of the system reliability. The parameters are measured by direct and indirect means, and upper bounds are estimated. A nonparametric method based on an asymptotic property of the tail of a distribution is used to estimate the upper bound of a critical system parameter. Although the proof process is very costly, it is extremely valuable when validating the crucial synchronization subsystem.

Johnson, S. C.↗

Validation of the F-18 high alpha research vehicle flight control and avionics systems modifications

The verification and validation process is a critical portion of the development of a flight system. Verification, the steps taken to assure the system meets the design specification, has become a reasonably understood and straightforward process. Validation is the method used to ensure that the system design meets the needs of the project. As systems become more integrated and more critical in their functions, the validation process becomes more complex and important. The tests, tools, and techniques which are being used for the validation of the high alpha research vehicle (HARV) turning vane control system (TVCS) are discussed and the problems and their solutions are documented. The emphasis of this paper is on the validation of integrated system.

Chacon, Vince↗

Validation environment for AIPS/ALS: Implementation and results

The work is presented which was performed in porting the Fault Injection-based Automated Testing (FIAT) and Programming and Instrumentation Environments (PIE) validation tools, to the Advanced Information Processing System (AIPS) in the context of the Ada Language System (ALS) application, as well as an initial fault free validation of the available AIPS system. The PIE components implemented on AIPS provide the monitoring mechanisms required for validation. These mechanisms represent a substantial portion of the FIAT system. Moreover, these are required for the implementation of the FIAT environment on AIPS. Using these components, an initial fault free validation of the AIPS system was performed. The implementation is described of the FIAT/PIE system, configured for fault free validation of the AIPS fault tolerant computer system. The PIE components were modified to support the Ada language. A special purpose AIPS/Ada runtime monitoring and data collection was implemented. A number of initial Ada programs running on the PIE/AIPS system were implemented. The instrumentation of the Ada programs was accomplished automatically inside the PIE programming environment. PIE's on-line graphical views show vividly and accurately the performance characteristics of Ada programs, AIPS kernel and the application's interaction with the AIPS kernel. The data collection mechanisms were written in a high level language, Ada, and provide a high degree of flexibility for implementation under various system conditions.

Segall, Zary↗

Sensor data validation and reconstruction. Phase 1: System architecture study

The sensor validation and data reconstruction task reviewed relevant literature and selected applicable validation and reconstruction techniques for further study; analyzed the selected techniques and emphasized those which could be used for both validation and reconstruction; analyzed Space Shuttle Main Engine (SSME) hot fire test data to determine statistical and physical relationships between various parameters; developed statistical and empirical correlations between parameters to perform validation and reconstruction tasks, using a computer aided engineering (CAE) package; and conceptually designed an expert system based knowledge fusion tool, which allows the user to relate diverse types of information when validating sensor data. The host hardware for the system is intended to be a Sun SPARCstation, but could be any RISC workstation with a UNIX operating system and a windowing/graphics system such as Motif or Dataviews. The information fusion tool is intended to be developed using the NEXPERT Object expert system shell, and the C programming language.

Source record↗