SEARCH · Search NASA
Results for “model skill”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Commercially-viable Hydrogen Aircraft for Reduction of Greenhouse Emissions
NASA assembled a cross-organizational multidisciplinary “radical” project team combining a diverse set of skills including aircraft architecture modelling and optimization, advanced material science, and engineering of high performance cryogenic, thermal management and fuel cell systems components and subsystems to tackle the challenging problem of development of commercially viable mid-size aircraft that would radically transform air transportation. Our team is developing an integrated conceptual and experimental methodology to realize a medium-range hydrogen aircraft design based on fuel cells, hydrogen burning engines, advanced power management and distribution, cryogenic hydrogen storage systems, and novel thermal management systems combined with an integrated aircraft concept of operations both during the flight and at the airports. The resulting analyses suggested the aircraft architecture options, sizes and layouts for the propulsion subsystem and cryogenic liquid hydrogen (LH2) tankage to verify the weight-scaling relationships for a medium-range aircraft carrying 100-200 passengers flying 1000 - 5000 km. Hydrogen - based distributed electric propulsion and cryogenic systems were further analysed, and more detailed study identified systems goals for a viable overall system weight for missions of various lengths. The developed aircraft architecture is being optimized by total specific energy density, specific power, size and mission profiles.
Commercially-viable Hydrogen Aircraft for Reduction of Greenhouse Emissions
NASA assembled a cross-organizational multidisciplinary “radical” project team combining a diverse set of skills including aircraft architecture modelling and optimization, advanced material science, and engineering of high performance cryogenic, thermal management and fuel cell systems components and subsystems to tackle the challenging problem of development of commercially viable mid-size aircraft that would radically transform air transportation. Our team is developing an integrated conceptual and experimental methodology to realize a medium-range hydrogen aircraft design based on fuel cells, hydrogen burning engines, advanced power management and distribution, cryogenic hydrogen storage systems, and novel thermal management systems combined with an integrated aircraft concept of operations both during the flight and at the airports. The resulting analyses suggested the aircraft architecture options, sizes and layouts for the propulsion subsystem and cryogenic liquid hydrogen (LH2) tankage to verify the weight-scaling relationships for a medium-range aircraft carrying 100-200 passengers flying 1000 - 5000 km. Hydrogen - based distributed electric propulsion and cryogenic systems were further analysed, and more detailed study identified systems goals for a viable overall system weight for missions of various lengths. The developed aircraft architecture is being optimized by total specific energy density, specific power, size and mission profiles.
Estimating Surface Soil Moisture from SMAP Observations Using a Neural Network Technique
A Neural Network (NN) algorithm was developed to estimate global surface soil moisture for April 2015 to March 2017 with a 2-3 day repeat frequency using passive microwave observations from the Soil Moisture Active Passive (SMAP) satellite, surface soil temperatures from the NASA Goddard Earth Observing System Model version 5 (GEOS-5) land modeling system, and Moderate Resolution Imaging Spectroradiometer-based vegetation water content. The NN was trained on GEOS-5 soil moisture target data, making the NN estimates consistent with the GEOS-5 climatology, such that they may ultimately be assimilated into this model without further bias correction. Evaluated against in situ soil moisture measurements, the average unbiased root mean square error (ubRMSE), correlation and anomaly correlation of the NN retrievals were 0.037 m(exp. 3)m(exp. -3), 0.70 and 0.66, respectively, against SMAP core validation site measurements and 0.026 m(exp. 3)m(exp. -3), 0.58 and 0.48, respectively, against International Soil Moisture Network (ISMN) measurements. At the core validation sites, the NN retrievals have a significantly higher skill than the GEOS-5 model estimates and a slightly lower correlation skill than the SMAP Level-2 Passive (L2P) product. The feasibility of the NN method was reflected by a lower ubRMSE compared to the L2P retrievals as well as a higher skill when ancillary parameters in physically-based retrievals were uncertain. Against ISMN measurements, the skill of the two retrieval products was more comparable. A triple collocation analysis against Advanced Microwave Scanning Radiometer 2 (AMSR2) and Advanced Scatterometer (ASCAT) soil moisture retrievals showed that the NN and L2P retrieval errors have a similar spatial distribution, but the NN retrieval errors are generally lower in densely vegetated regions and transition zones.
Impasse-driven tutoring for reactive skill acquisition
We are interested in developing effective performance-oriented training for the operation of systems that are used for monitor and control purposes. We have focused on one such system, the communications Link Monitor and Control (LMC) system used in NASA's Deep Space Network (DSN), which is a worldwide system for navigating, tracking and communicating with unmanned interplanetary spacecraft. The tasks in this domain are procedural in nature and require reactive, goal-oriented skills; we have previously described a cognitive model for problem solving that accounts for both novice and expert levels of behavior as well as how skill is acquired. Our cognitive modeling work in this task domain led us to make a number of predictions about tutoring that have influenced the design of the system described in this paper.
Effect of high latitude filtering on NWP skill
An assessment is made of the extent to which polar filtering may seriously affect the skill of latitude-longitude NWP models, such as the U.S. Navy's NOGAPS, or the GLAS fourth-order model. The limited experiments which have been completed to date with the 4 x 5-deg, 9-level version of the latter model indicate that the high latitude filter currently in operation affects its forecasting skill very little, with only one exception in which the use of the PG filter significantly improved forecasting.
Toward an embedded training tool for Deep Space Network operations
There are three issues to consider when building an embedded training system for a task domain involving the operation of complex equipment: (1) how skill is acquired in the task domain; (2) how the training system should be designed to assist in the acquisition of the skill, and more specifically, how an intelligent tutor could aid in learning; and (3) whether it is feasible to incorporate the resulting training system into the operational environment. This paper describes how these issues have been addressed in a prototype training system that was developed for operations in NASA's Deep Space Network (DSN). The first two issues were addressed by building an executable cognitive model of problem solving and skill acquisition of the task domain and then using the model to design an intelligent tutor. The cognitive model was developed in Soar for the DSN's Link Monitor and Control (LMC) system; it led to several insights about learning in the task domain that were used to design an intelligent tutor called REACT that implements a method called 'impasse-driven tutoring'. REACT is one component of the LMC training system, which also includes a communications link simulator and a graphical user interface. A pilot study of the LMC training system indicates that REACT shows promise as an effective way for helping operators to quickly acquire expert skills.
Toward an Embedded Training Tool for Deep Space Network Operations
There are three issues to consider when building an embedded training system for a task domain involving the operation of complex equipment: (1) how skill is acquired in the task domain; (2) how the training system should be designed to assist in the acquisition of the skill, and more specifically, how an intelligent tutor could aid in learning; and (3) whether it is feasible to incorporate the resulting training system into the operational environment. This paper describes how these issues have been addressed in a prototype training system that was developed for operations in NASA's Deep Space Network (DSN). The first two issues were addressed by building an executable cognitive model of problem solving and skill acquisition of the task domain and then using the model to design an intelligent tutor.
Investigating the Simulink Auto-Coding Process
Model based program design is the most clear and direct way to develop algorithms and programs for interfacing with hardware. While coding "by hand" results in a more tailored product, the ever-growing size and complexity of modern-day applications can cause the project work load to quickly become unreasonable for one programmer. This has generally been addressed by splitting the product into separate modules to allow multiple developers to work in parallel on the same project, however this introduces new potentials for errors in the process. The fluidity, reliability and robustness of the code relies on the abilities of the programmers to communicate their methods to one another; furthermore, multiple programmers invites multiple potentially differing coding styles into the same product, which can cause a loss of readability or even module incompatibility. Fortunately, Mathworks has implemented an auto-coding feature that allows programmers to design their algorithms through the use of models and diagrams in the graphical programming environment Simulink, allowing the designer to visually determine what the hardware is to do. From here, the auto-coding feature handles converting the project into another programming language. This type of approach allows the designer to clearly see how the software will be directing the hardware without the need to try and interpret large amounts of code. In addition, it speeds up the programming process, minimizing the amount of man-hours spent on a single project, thus reducing the chance of human error as well as project turnover time. One such project that has benefited from the auto-coding procedure is Ramses, a portion of the GNC flight software on-board Orion that has been implemented primarily in Simulink. Currently, however, auto-coding Ramses into C++ requires 5 hours of code generation time. This causes issues if the tool ever needs to be debugged, as this code generation will need to occur with each edit to any part of the program; additionally, this is lost time that could be spent testing and analyzing the code. This is one of the more prominent issues with the auto-coding process, and while much information is available with regard to optimizing Simulink designs to produce efficient and reliable C++ code, not much research has been made public on how to reduce the code generation time. It is of interest to develop some insight as to what causes code generation times to be so significant, and determine if there are architecture guidelines or a desirable auto-coding configuration set to assist in streamlining this step of the design process for particular applications. To address the issue at hand, the Simulink coder was studied at a foundational level. For each different component type made available by the software, the features, auto-code generation time, and the format of the generated code were analyzed and documented. Tools were developed and documented to expedite these studies, particularly in the area of automating sequential builds to ensure accurate data was obtained. Next, the Ramses model was examined in an attempt to determine the composition and the types of technologies used in the model. This enabled the development of a model that uses similar technologies, but takes a fraction of the time to auto-code to reduce the turnaround time for experimentation. Lastly, the model was used to run a wide array of experiments and collect data to obtain knowledge about where to search for bottlenecks in the Ramses model. The resulting contributions of the overall effort consist of an experimental model for further investigation into the subject, as well as several automation tools to assist in analyzing the model, and a reference document offering insight to the auto-coding process, including documentation of the tools used in the model analysis, data illustrating some potential problem areas in the auto-coding process, and recommendations on areas or practices in the current Ramses model that should be further investigated. Several skills were required to be built up over the course of the internship project. First and foremost, my Simulink skills have improved drastically, as much of my experience had been modeling electronic circuits as opposed to software models. Furthermore, I am now comfortable working with the Simulink Auto-coder, a tool I had never used until this summer; this tool also tested my critical thinking and C++ knowledge as I had to interpret the C++ code it was generating and attempt to understand how the Simulink model affected the generated code. I had come into the internship with a solid understanding of Matlab code, but had done very little in using it to automate tasks, particularly Simulink tasks; along the same lines, I had rarely used shell script to automate and interface with programs, which I gained a fair amount of experience with this summer, including how to use regular expression. Lastly, soft-skills are an area everyone can continuously improve on; having never worked with NASA engineers, which to me seem to be a completely different breed than what I am used to (commercial electronic engineers), I learned to utilize the wealth of knowledge present at JSC. I wish I had come into the internship knowing exactly how helpful everyone in my branch would be, as I would have picked up on this sooner. I hope that having gained such a strong foundation in Simulink over this summer will open the opportunity to return to work on this project, or potentially other opportunities within the division. The idea of leaving a project I devoted ten weeks to is a hard one to cope with, so having the chance to pick up where I left off sounds appealing; alternatively, I am interested to see if there are any opening in the future that would allow me to work on a project that is more in-line with my research in estimation algorithms. Regardless, this summer has been a milestone in my professional career, and I hope this has started a long-term relationship between JSC and myself. I really enjoy the thought of building on my experience here over future summers while I work to complete my PhD at Missouri University of Science and Technology.
An analysis of Space Shuttle countdown activities: Preliminaries to a computational model of the NASA Test Director
Before all systems are go just prior to the launch of a space shuttle, thousands of operations and tests have been performed to ensure that all shuttle and support subsystems are operational and ready for launch. These steps, which range from activating the orbiter's flight computers to removing the launch pad from the itinerary of the NASA tour buses, are carried out by launch team members at various locations and with highly specialized fields of expertise. The liability for coordinating these diverse activities rests with the NASA Test Director (NTD) at NASA-Kennedy. The behavior is being studied of the NTD with the goal of building a detailed computational model of that behavior; the results of that analysis to date are given. The NTD's performance is described in detail, as a team member who must coordinate a complex task through efficient audio communication, as well as an individual taking notes and consulting manuals. A model of the routine cognitive skill used by the NTD to follow the launch countdown procedure manual was implemented using the Soar cognitive architecture. Several examples are given of how such a model could aid in evaluating proposed computer support systems.
Machine learning methods for weather forecasting
SAND2025-14466O This repository contains code for developing, training, and evaluating machine learning models for weather and climate forecasting, including forecast skill assessment, feature importance analysis, and reproducible workflows for model comparison. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
Using Apex To Construct CPM-GOMS Models
process for automatically generating computational models of human/computer interactions as well as graphical and textual representations of the models has been built on the conceptual foundation of a method known in the art as CPM-GOMS. This method is so named because it combines (1) the task decomposition of analysis according to an underlying method known in the art as the goals, operators, methods, and selection (GOMS) method with (2) a model of human resource usage at the level of cognitive, perceptual, and motor (CPM) operations. CPM-GOMS models have made accurate predictions about behaviors of skilled computer users in routine tasks, but heretofore, such models have been generated in a tedious, error-prone manual process. In the present process, CPM-GOMS models are generated automatically from a hierarchical task decomposition expressed by use of a computer program, known as Apex, designed previously to be used to model human behavior in complex, dynamic tasks. An inherent capability of Apex for scheduling of resources automates the difficult task of interleaving the cognitive, perceptual, and motor resources that underlie common task operators (e.g., move and click mouse). The user interface of Apex automatically generates Program Evaluation Review Technique (PERT) charts, which enable modelers to visualize the complex parallel behavior represented by a model. Because interleaving and the generation of displays to aid visualization are automated, it is now feasible to construct arbitrarily long sequences of behaviors. The process was tested by using Apex to create a CPM-GOMS model of a relatively simple human/computer-interaction task and comparing the time predictions of the model and measurements of the times taken by human users in performing the various steps of the task. The task was to withdraw $80 in cash from an automated teller machine (ATM). For the test, a Visual Basic mockup of an ATM was created, with a provision for input from (and measurement of the performance of) the user via a mouse. The times predicted by the automatically generated model turned out to approximate the measured times fairly well (see figure). While these results are promising, there is need for further development of the process. Moreover, it will also be necessary to test other, more complex models: The actions required of the user in the ATM task are too sequential to involve substantial parallelism and interleaving and, hence, do not serve as an adequate test of the unique strength of CPM-GOMS models to accommodate parallelism and interleaving.
Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction
Abstract The rapid rise of deep learning (DL) in numerical weather prediction (NWP) has led to a proliferation of models which forecast atmospheric variables with comparable or superior skill than traditional physics-based NWP. However, among these leading DL models, there is a wide variance in both the training settings and architecture used. Further, the lack of thorough ablation studies makes it hard to discern which components are most critical to success. In this work, we show that it is possible to attain high forecast skill even with relatively off-the-shelf architectures, simple training procedures, and moderate compute budgets. Specifically, we train a minimally modified Swin Transformer V2 (SwinV2) on ERA5 data and find that it attains superior skill in terms of mean-square errors of deterministic forecasts when compared against the European Centre for Medium-Range Weather Forecasts’ Integrated Forecasting System (IFS). Almost all DL–NWP systems share a core set of hyperparameters and design decisions. To aid and expedite future DL–NWP research, we present an in-depth, systematic exploration of different loss functions, model sizes and depths, patch sizes, and multistep training objectives. We also examine the model performance with metrics beyond the typical accuracy (ACC) and RMSE and investigate how the performance scales with model size. Through our open-source code, scoring pipelines, and models, we share our findings on key aspects of the training pipeline. These ablations reduce the necessity for expensive hyperparameter tuning and lower the barrier to entry for future DL–NWP research. Significance Statement This study investigates the potential of using large-scale transformer-based models for weather prediction, showing that it is possible to achieve high forecast accuracy with simpler, off-the-shelf architectures. By training a minimally modified SwinV2 transformer on ERA5 data, we show that the model achieves competitive forecast skill in terms of mean-square error for key variables, outperforming the European Centre for Medium-Range Weather Forecasts’ Integrated Forecasting System (IFS) at all lead times. Our findings suggest that effective training strategies, such as multistep fine-tuning and channel-weighted losses, significantly enhance the model’s performance. However, we also highlight that these improvements come with trade-offs in other areas, such as ensemble spread and high-frequency spatial detail. This work highlights the promise of deep learning in improving weather forecasts, which could lead to better preparedness and response to weather events, ultimately benefiting society by providing more reliable weather predictions.
Comprehensive Assessment of Models and Events Using Library Tools (CAMEL) Framework: Time Series Comparisons
The Comprehensive Assessment of Models and Events using Library Tools (CAMEL) framework leverages existing Community Coordinated Modeling Center services: Run on Request post processing tools that generate model time series outputs and the new Community Coordinated Modeling Center Metadata Registry that describes simulation runs using Space Physics Archive Search and Extract metadata. The new CAMEL visualization tool compares the modeled time series with observational data and computes a suite of skill scores such as Prediction Efficiency, Root Mean Square Error, and Symmetric Signed Percentage Bias. Model data pairs used for skill calculations are obtained considering a user selected maximum difference between the time of observation and the nearest model output. The system renders available data for all locations and time periods selected using interactive visualizations that allow the user to zoom, pan, and pick data values along traces. Skill scores are reported for each selected event or aggregated over all events for all participating model runs. Separately, scores are reported for all locations (satellites or stations) and for each location individually. We are building on past experiences with model data comparisons of magnetosphere and ionosphere model outputs from GEM2008, GEMCEDAR Electrodynamics Thermosphere Ionosphere, and the SWPC Operational Space Weather Model challenges. The CAMEL visualization tool is demonstrated using three validation studies: (a) Wang Sheeley Arge heliosphere simulations compared against OMNI solar wind data, (b) ground magnetic perturbations from several magnetosphere and ionosphere electrodynamics models as observed by magnetometers, and (c) electron fluxes from several ring current simulations compared to Radiation Belt Storm Probes Helium Oxygen Proton Electron instrument measurements, integrated over different energy ranges.
Coupled Data Assimilation for Integrated Earth System Analysis and Prediction: Goals, Challenges, and Recommendations
The purpose of this report is to identify fundamental issues for coupled data assimilation (CDA), such as gaps in science and limitations in forecasting systems, in order to provide guidance to the World Meteorological Organization (WMO) on how to facilitate more rapid progress internationally. Coupled Earth system modeling provides the opportunity to extend skillful atmospheric forecasts beyond the traditional two-week barrier by extracting skill from low-frequency state components such as the land, ocean, and sea ice. More generally, coupled models are needed to support seamless prediction systems that span timescales from weather, subseasonal to seasonal (S2S), multiyear, and decadal. Therefore, initialization methods are needed for coupled Earth system models, either applied to each individual component (called Weakly Coupled Data Assimilation - WCDA) or applied the coupled Earth system model as a whole (called Strongly Coupled Data Assimilation - SCDA). Using CDA, in which model forecasts and potentially the state estimation are performed jointly, each model domain benefits from observations in other domains either directly using error covariance information known at the time of the analysis (SCDA), or indirectly through flux interactions at the model boundaries (WCDA). Because the non-atmospheric domains are generally under-observed compared to the atmosphere, CDA provides a significant advantage over single-domain analyses. Next, we provide a synopsis of goals, challenges, and recommendations to advance CDA: Goals: (a) Extend predictive skill beyond the current capability of NWP (e.g. as demonstrated by improving forecast skill scores), (b) produce physically consistent initial conditions for coupled numerical prediction systems and reanalyses (including consistent fluxes at the domain interfaces), (c) make best use of existing observations by allowing observations from each domain to influence and improve the full earth system analysis, (d) develop a robust observation-based identification and understanding of mechanisms that determine the variability of weather and climate, (e) identify critical weaknesses in coupled models and the earth observing system, (f) generate full-field estimates of unobserved or sparsely observed variables, (g) improve the estimation of the external forcings causing changes to climate, (h) transition successes from idealized CDA experiments to real-world applications. Challenges: (a) Modeling at the interfaces between interacting components of coupled Earth system models may be inadequate for estimating uncertainty or error covariances between domains, (b) current data assimilation methods may be insufficient to simultaneously analyze domains containing multiple spatiotemporal scales of interest, (c) there is no standardization of observation data or their delivery systems across domains, (d) the size and complexity of many large-scale coupled Earth system models makes it is difficult to accurately represent uncertainty due to model parameters and coupling parameters, (e) model errors lead to local biases that can transfer between the different Earth system components and lead to coupled model biases and long-term model drift, (e) information propagation across model components with different spatiotemporal scales is extremely complicated, and must be improved in current coupled modeling frameworks, (h) there is insufficient knowledge on how to represent evolving errors in non-atmospheric model components (e.g. as sea ice, land and ocean) on the timescales of NWP.
Optimization of a Radiative Transfer Forward Operator for Simulating SMOS Brightness Temperatures over the Upper Mississippi Basin, USA
The Soil Moisture and Ocean Salinity (SMOS) satellite mission is routinely providing global multi-angular observations of brightness temperature (TB) at both horizontal and vertical polarization with a 3-day repeat period. The assimilation of such data into a land surface model (LSM) may improve the skill of operational flood forecasts through an improved estimation of soil moisture (SM). To accommodate for the direct assimilation of the SMOS TB data, the LSM needs to be coupled with a radiative transfer model (RTM), serving as a forward operator for the simulation of multi-angular and multi-polarization top of atmosphere TBs. This study investigates the use of the Variable Infiltration Capacity (VIC) LSM coupled with the Community Microwave Emission Modelling platform (CMEM) for simulating SMOS TB observations over the Upper Mississippi basin, USA. For a period of 2 years (2010-2011), a comparison between SMOS TBs and simulations with literature-based RTM parameters reveals a basin averaged bias of 30K. Therefore, time series of SMOS TB observations are used to investigate ways for mitigating these large biases. Specifically, the study demonstrates the impact of the LSM soil moisture climatology in the magnitude of TB biases. After CDF matching the SM climatology of the LSM to SMOS retrievals, the average bias decreases from 30K to less than 5K. Further improvements can be made through calibration of RTM parameters related to the modeling of surface roughness and vegetation. Consequently, it can be concluded that SM rescaling and RTM optimization are efficient means for mitigating biases and form a necessary preparatory step for data assimilation.