Search NASA⌕ Search

SEARCH · Search NASA

Results for “model skill”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
NASA NTRS1997

The Design of Model-Based Training Programs

This paper proposes a model-based training program for the skills necessary to operate advance avionics systems that incorporate advanced autopilots and fight management systems. The training model is based on a formalism, the operational procedure model, that represents the mission model, the rules, and the functions of a modem avionics system. This formalism has been defined such that it can be understood and shared by pilots, the avionics software, and design engineers. Each element of the software is defined in terms of its intent (What?), the rationale (Why?), and the resulting behavior (How?). The Advanced Computer Tutoring project at Carnegie Mellon University has developed a type of model-based, computer aided instructional technology called cognitive tutors. They summarize numerous studies showing that training times to a specified level of competence can be achieved in one third the time of conventional class room instruction. We are developing a similar model-based training program for the skills necessary to operation the avionics. The model underlying the instructional program and that simulates the effects of pilots entries and the behavior of the avionics is based on the operational procedure model. Pilots are given a series of vertical flightpath management problems. Entries that result in violations, such as failure to make a crossing restriction or violating the speed limits, result in error messages with instruction. At any time, the flightcrew can request suggestions on the appropriate set of actions. A similar and successful training program for basic skills for the FMS on the Boeing 737-300 was developed and evaluated. The results strongly support the claim that the training methodology can be adapted to the cockpit.

Polson, Peter↗
DOE OSTI2025

A Practical Probabilistic Benchmark for AI Weather Models

Since the weather is chaotic, it is necessary to forecast an ensemble of future states. Recently, multiple AI weather models have emerged claiming breakthroughs in deterministic skill. Unfortunately, it is hard to fairly compare ensembles of AI forecasts because variations in ensembling methodology become confounding and the baseline data volume is immense. We address this by scoring lagged initial condition ensembles—whereby an ensemble can be constructed from a library of deterministic hindcasts. This allows the first parameter‐free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. Lagged ensembles of the two leading AI weather models, GraphCast and Pangu, perform similarly even though the former outperforms the latter in deterministic scoring. These results are elaborated upon by sensitivity tests showing that commonly used multiple time‐step loss functions damage ensemble calibration.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1977

The effect of increased horizontal resolution on synoptic forecasts with the GISS model of the global atmosphere

The improved horizontal resolution global circulation model in question has a horizontal spacing of 2.5 deg latitude by 3 deg longitude for a resolution of about 250 km in midlatitude. This paper reports on experiments to determine the improvements in forecasting skill and in initial conditions that were made possible by the ultrafine resolution and ultrafine assimilation. The size of the improvement in skill score when the ultrafine model is used for forecasting and assimilation was so large that 60-hour forecasts with the ultrafine model had as good a skill score as the 48-hour forecasts with the fine model. Synoptic evaluations confirmed that ultrafine model forecasts are better than the fine model forecasts.

Quirk, W. J.↗
NASA NTRS2002

CATS-based Air Traffic Controller Agents

This report describes intelligent agents that function as air traffic controllers. Each agent controls traffic in a single sector in real time; agents controlling traffic in adjoining sectors can coordinate to manage an arrival flow across a given meter fix. The purpose of this research is threefold. First, it seeks to study the design of agents for controlling complex systems. In particular, it investigates agent planning and reactive control functionality in a dynamic environment in which a variety perceptual and decision making skills play a central role. It examines how heuristic rules can be applied to model planning and decision making skills, rather than attempting to apply optimization methods. Thus, the research attempts to develop intelligent agents that provide an approximation of human air traffic controller behavior that, while not based on an explicit cognitive model, does produce task performance consistent with the way human air traffic controllers operate. Second, this research sought to extend previous research on using the Crew Activity Tracking System (CATS) as the basis for intelligent agents. The agents use a high-level model of air traffic controller activities to structure the control task. To execute an activity in the CATS model, according to the current task context, the agents reference a 'skill library' and 'control rules' that in turn execute the pattern recognition, planning, and decision-making required to perform the activity. Applying the skills enables the agents to modify their representation of the current control situation (i.e., the 'flick' or 'picture'). The updated representation supports the next activity in a cycle of action that, taken as a whole, simulates air traffic controller behavior. A third, practical motivation for this research is to use intelligent agents to support evaluation of new air traffic control (ATC) methods to support new Air Traffic Management (ATM) concepts. Current approaches that use large, human-in-the-loop simulations are unquestionably valuable for this purpose, but pose considerable logistical, fiscal, and experimental control problems. First, data analysis is extremely complicated, owing simply to the large number of participants and data sources in such simulations. In addition, experienced human air traffic controllers working adjacent sectors tend to flexibly adapt to the evolving control problem - potentially shifting to other strategies than those under investigation. In addition, their performance is tightly coupled to the control interface, which in the development phase may support some concepts and supporting strategies better than others. A simple shift in strategy by one controller can change the character of a particular traffic scenario dramatically, which makes experimental comparison of ATC performance under different traffic scenarios difficult. Training a given team of controllers on operations under a new ATM concept for a sufficient period of time could avert such difficulties, but instituting an adequate training program is expensive and logistically difficult.

Callantine, Todd J.↗
NASA NTRS1982

The effect of model resolution and satellite sounding data on GLAS model forecasts

The effect of horizontal model resolution on satellite data impact has been studied for two versions of the GLAS second-order general circulation model: the C-model with a 4-deg latitude by 5-deg longitude resolution and the F-model with a 2.5-deg latitude and 3-deg longitude resolution. It is found that the 48-72 h forecast skill of the GLAS model was significantly improved by the increased resolution. Initial state differences between the SAT and NOSAT cycles using the F-model were on the average smaller than the corresponding differences with the C-model. However, the F-model cycle differences exhibited a smaller scale structure and, in some cases, larger gradients.

Atlas, R.↗
NASA NTRS2002

Automating CPM-GOMS

CPM-GOMS is a modeling method that combines the task decomposition of a GOMS analysis with a model of human resource usage at the level of cognitive, perceptual, and motor operations. CPM-GOMS models have made accurate predictions about skilled user behavior in routine tasks, but developing such models is tedious and error-prone. We describe a process for automatically generating CPM-GOMS models from a hierarchical task decomposition expressed in a cognitive modeling tool called Apex. Resource scheduling in Apex automates the difficult task of interleaving the cognitive, perceptual, and motor resources underlying common task operators (e.g. mouse move-and-click). Apex's UI automatically generates PERT charts, which allow modelers to visualize a model's complex parallel behavior. Because interleaving and visualization is now automated, it is feasible to construct arbitrarily long sequences of behavior. To demonstrate the process, we present a model of automated teller interactions in Apex and discuss implications for user modeling. available to model human users, the Goals, Operators, Methods, and Selection (GOMS) method [6, 21] has been the most widely used, providing accurate, often zero-parameter, predictions of the routine performance of skilled users in a wide range of procedural tasks [6, 13, 15, 27, 28]. GOMS is meant to model routine behavior. The user is assumed to have methods that apply sequences of operators and to achieve a goal. Selection rules are applied when there is more than one method to achieve a goal. Many routine tasks lend themselves well to such decomposition. Decomposition produces a representation of the task as a set of nested goal states that include an initial state and a final state. The iterative decomposition into goals and nested subgoals can terminate in primitives of any desired granularity, the choice of level of detail dependent on the predictions required. Although GOMS has proven useful in HCI, tools to support the construction of GOMS models have not yet come into general use.

GOMS↗
NASA NTRS2023

Impact of Design Factors for ESA CCI Satellite Soil Moisture Data Assimilation over Europe

In this study, soil moisture retrievals of the combined active–passive ESA Climate Change Initiative (CCI) soil moisture product are assimilated into the Noah-MP land surface model over Europe using a one-dimensional ensemble Kalman filter and an 18-yr study period. The performance of the data assimilation (DA) system is evaluated by comparing it with a model-only experiment (at in situ sites) and by assessing statistics of innovations and increments as DA diagnostics (over the entire domain). For both assessments, we explore the impact of three design choices, resulting in the following insights. 1) The magnitude of the assumed observation errors strongly affects the skill improvements evaluated against in situ stations and internal diagnostics. 2) Choosing between climatological or monthly cumulative distribution function matching as the observation bias correction method only has a marginal effect on the in situ skill of the DA system. However, the internal diagnostics suggest a more robust system parameterization if the observations are rescaled monthly. 3) The choice of atmospheric reanalysis dataset to force the land surface model affects the model-only skill and the DA skill improvements. The model-only skill is higher with input from the MERRA-2 than with input from the ERA5 reanalysis, resulting in larger DA skill improvements for the latter. Additionally, we show that the added value of the DA strongly depends on the quality of the satellite retrievals and land cover, with the most substantial soil moisture skill improvements occurring over croplands and skill degradation occurring over densely forested areas.

Soil moisture↗
NASA NTRS2018

Machine Learning Application to Atmospheric Chemistry Modeling

Atmospheric chemistry is a high-dimensionality, large-data problem and thus may be suited to machine-learning algorithms. We show here the potential of a random forest regression algorithm to replace the gas-phase chemistry solver in the GEOS-Chem chemistry model. In this proof-of-concept study, we used one month of model output to train random forest regression models to predict the concentrations of each long-lived chemical species after integration based upon the physical and chemical conditions before the chemical integration. The choice of prediction type has a strong impact on the skill of the regression model. We find best results from predicting the change in concentration for very long-lived species and the absolute concentration for shorter lived species. The skill of the machine learning algorithm is further improved by using a family approach for NO and NO2 rather than treating them independently.By replacing the numerical integrator with the random forest algorithm and running this model for one month, we find that the model is able to reproduce many of the features of the reference chemistry simulation. Replacing the integration methodology with a machine learning algorithm has the potential to be substantially faster. There are a wide range of applications for such an approach, e.g. to generate boundary conditions, for use in air quality forecasts or chemical data assimilation systems, etc.

Keller, Christoph A.↗
NASA NTRS2016

Assessment of Arctic and Antarctic Sea Ice Predictability in CMIP5 Decadal Hindcasts

This paper examines the ability of coupled global climate models to predict decadal variability of Arctic and Antarctic sea ice. We analyze decadal hindcasts/predictions of 11 Coupled Model Intercomparison Project Phase 5 (CMIP5) models. Decadal hindcasts exhibit a large multimodel spread in the simulated sea ice extent, with some models deviating significantly from the observations as the predicted ice extent quickly drifts away from the initial constraint. The anomaly correlation analysis between the decadal hindcast and observed sea ice suggests that in the Arctic, for most models, the areas showing significant predictive skill become broader associated with increasing lead times. This area expansion is largely because nearly all the models are capable of predicting the observed decreasing Arctic sea ice cover. Sea ice extent in the North Pacific has better predictive skill than that in the North Atlantic (particularly at a lead time of 3-7 years), but there is a reemerging predictive skill in the North Atlantic at a lead time of 6-8 years. In contrast to the Arctic, Antarctic sea ice decadal hindcasts do not show broad predictive skill at any timescales, and there is no obvious improvement linking the areal extent of significant predictive skill to lead time increase. This might be because nearly all the models predict a retreating Antarctic sea ice cover, opposite to the observations. For the Arctic, the predictive skill of the multi-model ensemble mean outperforms most models and the persistence prediction at longer timescales, which is not the case for the Antarctic. Overall, for the Arctic, initialized decadal hindcasts show improved predictive skill compared to uninitialized simulations, although this improvement is not present in the Antarctic.

sea ice↗
NASA NTRS2018

Prediction Skill of U.S. Flash Droughts in Subseasonal Experiment (SubX) Model Hindcasts

Droughts that establish themselves over a short period of time (weeks to a few months), referred to as flash droughts, can have devastating impacts on agriculture, water resources, and ecosystems. The ability to predict such droughts in advance would greatly enhance our preparation for them and potentially reduce their impacts. The sub-seasonal time scale at which flash droughts occur emphasizes the importance of producing forecasts at weekly or finer intervals that extend beyond the numerical weather prediction time frame. Here we assess the ability of eight global forecast systems, each participating in the Sub-seasonal Experiment project (SubX), to predict key features associated with rapidly developing droughts over the United States during the last two decades. MERRA2 reanalysis is used as observations. Prediction skill for temperature and precipitation anomalies during these events is limited to the first 1-2 weeks after initialization for most hindcasts. However, there are some hindcasts in which large anomalies are well predicted 3-4 weeks or more in advance. The physical mechanisms that are key to the development of surface anomalies, including quasi-stationary atmospheric waves, were also evaluated. Most hindcasts were unable to capture the development or progression of such drought-inducing circulation features more than 1-2 weeks in advance.

Deangelis, Anthony M.↗
NASA NTRS2020

Using a Simple Water Balance Framework to Quantify the Impact of Soil Moisture Initialization on Subseasonal Evapotranspiration and Air Temperature Forecasts

Past studies have shown that accurate soil moisture initialization can contribute significant skill to near-surface air temperature (T2M) forecasts at subseasonal leads. The mechanisms by which soil moisture contributes such skill are examined here with a simple water balance-based model that captures the essence of soil moisture behavior in a state-of-the-art subseasonal-to-seasonal (S2S) forecasting system. The simple model successfully transforms initial soil moisture contents into average “forecasted” ET values at 16-30 day lead that agree well, during summer, with the values forecasted by the full NASA GEOS S2S system, indicating that soil moisture initialization dominates over forecasted meteorology in determining ET fluxes at subseasonal leads. When the simple model’s ET anomalies are interpreted in terms of T2M anomalies, a similar conclusion is reached for T2M: soil moisture initialization explains much (about 50% in the eastern half of the continental US) of the T2M anomalies produced by the full GEOS S2S system at 16-30 day lead, and the T2M forecasts produced by the simple model capture about half of the skill attained by the full system. The simple model’s framework is particularly conducive to an analysis of uncertainty in forecasts. Drier soils are generally found to induce larger uncertainty in ET (and thus T2M) forecasts, a result linked to the functional form relating ET to soil moisture in the simple model and verified by an analysis of the ensemble spreads within the forecasts produced by the full GEOS S2S system

Randal D Koster↗
NASA NTRS1990

Space-based Doppler lidar sampling strategies: Algorithm development and simulated observation experiments

Lidar Atmospheric Wind Sounder (LAWS) Simulation Models (LSM) were developed to evaluate the potential impact of global wind observations on the basic understanding of the Earth's atmosphere and on the predictive skills of current forecast models (GCM and regional scale). Fully integrated top to bottom LAWS Simulation Models for global and regional scale simulations were developed. The algorithm development incorporated the effects of aerosols, water vapor, clouds, terrain, and atmospheric turbulence into the models. Other additions include a new satellite orbiter, signal processor, line of sight uncertainty model, new Multi-Paired Algorithm and wind error analysis code. An atmospheric wind field library containing control fields, meteorological fields, phenomena fields, and new European Center for Medium Range Weather Forecasting (ECMWF) data was also added. The LSM was used to address some key LAWS issues and trades such as accuracy and interpretation of LAWS information, data density, signal strength, cloud obscuration, and temporal data resolution.

Emmitt, G. D.↗
NASA NTRS2020

The Boreal Winter El Niño Precipitation Response over North America: Insights into Why January is More Difficult to Predict than February

This study examines the within−season monthly variation of the El Niño response over North America during December−March using the NASA/GEOS model. In agreement with previous studies, the skill of 1−month lead GEOS coupled model forecasts of precipitation over North America is largest (smallest) for February (January), with similar results in uncoupled mode. A key finding is that the relatively poor January skillis the result ofthe model placing the main circulation anomaly over the northeast Pacific slightly to the west of the observed, resulting inprecipitation anomalies that lie off the coast instead of over land as observed. In contrast, during February the observed circulation anomaly over the northeast Pacific shifts westward, lining up with the predicted anomaly which is essentially unchanged from January, resulting in both the observed and predicted precipitation anomalies remaining off the coast. Furthermore, the largest precipitation anomalies occur along the southern tier of states associated with an eastward extended jet–something that the models capture reasonably well. Simulations with a stationary wave model indicate that the placement of January El Niño response to the west of the observed over the northeast Pacificis the result of biases in the January climatological stationary waves, rather than errors in the tropical Pacific El Niño heating anomalies in January. Furthermore, evidence is provided that the relatively poor simulation of the observed January climatology, characterized by a strengthened North Pacific jet and enhanced ridge over western North America, can be traced back to biases in the January climatology heating over the Tibet region and the tropical western Pacific.

Young-kwon Lim↗
NASA NTRS2013

Evaluating NMME Seasonal Forecast Skill for use in NASA SERVIR Hub Regions

The U.S. National Multi-Model Ensemble seasonal forecasting system is providing hindcast and real-time data streams to be used in assessing and improving seasonal predictive capacity. The coupled forecasts have numerous potential applications, both national and international in scope. The NASA / USAID SERVIR project, which leverages satellite and modeling-based resources for environmental decision making in developing nations, is focusing on the evaluation of NMME forecasts specifically for use in driving applications models in hub regions including East Africa, the Hindu Kush- Himalayan (HKH) region and Mesoamerica. A prerequisite for seasonal forecast use in application modeling (e.g. hydrology, agriculture) is bias correction and skill assessment. Efforts to address systematic biases and multi-model combination in support of NASA SERVIR impact modeling requirements will be highlighted. Specifically, quantilequantile mapping for bias correction has been implemented for all archived NMME hindcasts. Both deterministic and probabilistic skill estimates for raw, bias-corrected, and multi-model ensemble forecasts as a function of forecast lead will be presented for temperature and precipitation. Complementing this statistical assessment will be case studies of significant events, for example, the ability of the NMME forecasts suite to anticipate the 2010/2011 drought in the Horn of Africa and its relationship to evolving SST patterns.

Roberts, J. Brent↗
NASA NTRS2010

Role of Subsurface Physics in the Assimilation of Surface Soil Moisture Observations

Root zone soil moisture controls the land-atmosphere exchange of water and energy and exhibits memory that may be useful for climate prediction at monthly scales. Assimilation of satellite-based surface soil moisture observations into a land surface model is an effective way to estimate large-scale root zone soil moisture. The propagation of surface information into deeper soil layers depends on the model-specific representation of subsurface physics that is used in the assimilation system. In a suite of experiments we assimilate synthetic surface soil moisture observations into four different models (Catchment, Mosaic, Noah and CLM) using the Ensemble Kalman Filter. We demonstrate that identical twin experiments significantly overestimate the information that can be obtained from the assimilation of surface soil moisture observations. The second key result indicates that the potential of surface soil moisture assimilation to improve root zone information is higher when the surface to root zone coupling is stronger. Our experiments also suggest that (faced with unknown true subsurface physics) overestimating surface to root zone coupling in the assimilation system provides more robust skill improvements in the root zone compared with underestimating the coupling. When CLM is excluded from the analysis, the skill improvements from using models with different vertical coupling strengths are comparable for different subsurface truths. Finally, the skill improvements through assimilation were found to be sensitive to the regional climate and soil types.

Reichle, R. H.↗
NASA NTRS1991

Requirements for psychological models to support design: Towards ecological task analysis

Cognitive engineering is largely concerned with creating environmental designs to support skillful and effective human activity. A set of necessary conditions are proposed for psychological models capable of supporting this enterprise. An analysis of the psychological nature of the design product is used to identify a set of constraints that models must meet if they can usefully guide design. It is concluded that cognitive engineering requires models with resources for describing the integrated human-environment system, and that these models must be capable of describing the activities underlying fluent and effective interaction. These features are required in order to be able to predict the cognitive activity that will be required given various design concepts, and to design systems that promote the acquisition of fluent, skilled behavior. These necessary conditions suggest that an ecological approach can provide valuable resources for psychological modeling to support design. Relying heavily on concepts from Brunswik's and Gibson's ecological theories, ecological task analysis is proposed as a framework in which to predict the types of cognitive activity required to achieve productive behavior, and to suggest how interfaces can be manipulated to alleviate certain types of cognitive demands. The framework is described in terms, and illustrated with an example from the previous research on modeling skilled human-environment interaction.

Kirlik, Alex↗
DOE OSTI2025

Leveraging High-resolution Molecular Composition of Soil Organic Matter to Enhance Carbon Cycling Modeling

Soils store more carbon than the atmosphere and vegetation combined, yet Earth system models still struggle to predict how this vast reservoir will respond to environmental change. A central limitation is that most soil biogeochemical models represent organic matter using bulk conceptual pools or chemically homogeneous fractions, preventing direct use of rapidly expanding molecular-scale datasets. Here we develop and test a new soil decomposition framework that explicitly integrates high-resolution information on organic matter composition. First, we construct a molecularly informed litter decomposition module in which plant inputs are partitioned into five functional compound classes—carbohydrates, proteins, lignin-like aromatics, lipids, and carbonyls—using a molecular mixing model calibrated to solid-state 13 C Nuclear Magnetic Resonance (NMR) spectra. Class-specific kinetics, lignin-dependent physical protection, and substrate-driven microbial carbon use efficiency allow the module to capture metabolic tradeoffs associated with enzyme production and nutrient limitation. We then embed this litter module within a microbially explicit whole-soil model that tracks the transformation of these compound classes through particulate organic matter, dissolved organic matter, mineral-associated organic matter, and microbial biomass. High-resolution Fourier Transform Ion Cyclotron Resonance mass spectrometry (FTICR-MS) data are used to link internal pools to measurable soil organic matter fractions and to constrain key process parameters. Applications at soil-core and ecosystem scales demonstrate that the new model reproduces observed soil respiration dynamics while providing mechanistic attribution of CO 2 fluxes to specific chemical classes and pools. Compared to existing frameworks such as the Community Land Model soil biogeochemistry module and the Millennial model, our approach maintains competitive predictive skill while substantially improving interpretability and opportunities for data–model integration. This work illustrates a viable pathway for leveraging molecular-scale observations to reduce structural uncertainty in soil carbon–climate feedback projections.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2009

Skill Assessment of a Spectral Ocean-Atmosphere Radiative Model

Ocean phytoplankton, detrital material, and water absorb and scatter light spectrally. The Ocean- Atmosphere Spectral Irradiance Model (OASIM) is intended to provide surface irradiance over the oceans with sufficient spectral resolution to support ocean ecology, biogeochemistry, and heat exchange investigations, and of sufficient duration to support inter-annual and decadal investigations. OASIM total surface irradiance (integrated 200 nm to 4 microns) was compared to in situ data and three publicly available global data products at monthly 1-degree resolution. OASIM spectrally-integrated surface irradiance had root mean square (RMS) difference= 20.1 W/sq m (about 11%), bias=1.6 W/sq m (about 0.8%), regression slope= 1.01 and correlation coefficient= 0.89, when compared to 2322 in situ observations. OASIM had the lowest bias of any of the global data products evaluated (ISCCP-FD, NCEP, and ISLSCP 11), and the best slope (nearest to unity). It had the second best RMS, and the third best correlation coefficient. OASIM total surface irradiance compared well with ISCCP-FD (RMS= 20.7 W/sq m; bias=-11.4 W/sq m, r=0.98) and ISLSCP II (RMS =25.2 W/sq m; bias= -13.8 W/sq m; r=0.97), but less well with NCEP (RMS =43.0 W/sq m ;bias=-22.6 W/sq m; x=0.91). Comparisons of OASIM photosynthetically available radiation (PAR) with PAR derived from SeaWiFS showed low bias (-1.8 mol photons /sq m/d, or about 5%), RMS (4.25 mol photons /sq m/d ' or about 12%), near unity slope (1.03) and high correlation coefficient (0.97). Coupled with previous estimates of clear sky spectral irradiance in OASIM (6.6% RMS at 1 nm resolution), these results suggest that OASIM provides reasonable estimates of surface broadband and spectral irradiance in the oceans, and can support studies on ocean ecosystems, carbon cycling, and heat exchange.

Gregg, Watson, W.↗