Search NASA⌕ Search

SEARCH · Search NASA

Results for “Distributed training”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,189 records · Page 66

Multilabel proportion prediction and out-of-distribution detection on gamma spectra of short-lived fission products

In the machine learning problem of multilabel classification, the objective is to determine for each test instance which classes the instance belongs to. In this work, we consider an extension of multilabel classification, called multilabel proportion prediction, in the context of radioisotope identification (RIID) using gamma spectra data. We aim to not only predict radioisotope proportions, but also identify out-of-distribution (OOD) spectra. We achieve this goal by viewing gamma spectra as discrete probability distributions, and based on this perspective, we develop a custom semi-supervised loss function that combines a traditional supervised loss with an unsupervised reconstruction error function. Our approach was motivated by its application to the analysis of short-lived fission products from spent nuclear fuel. In particular, we demonstrate that a neural network model trained with our loss function can successfully predict the relative proportions of 37 radioisotopes simultaneously. The model trained with synthetic data was then applied to measurements taken by Pacific Northwest National Laboratory (PNNL) to conduct analysis typically done by subject-matter experts. Here, we also extend our approach to successfully identify when measurements are OOD, and thus should not be trusted, whether due to the presence of a novel source or novel proportions.

Anomaly detection↗

Inverse design of two-dimensional graphene/h-BN hybrids by a regressional and conditional GAN

Design of materials with desired properties is currently laborious and heavily relies on intuition of researchers through a trial-and-error process. To tackle this challenge, in this work we propose a novel regressional and conditional generative adversarial network (RCGAN) for inverse design of representative two-dimensional materials, the graphene and boron-nitride (BN) hybrids. RCGAN incorporates a supervised regressor network, thus overcoming the common technical barrier in the traditional unsupervised GANs, which cannot generate data when fed with continuous and quantitative labels. RCGAN can autonomously generate graphene/BN hybrids given any target bandgap values. These structures are distinguished from the ones used for training and exhibit high diversity for a given bandgap. Moreover, they exhibit high fidelity, yielding bandgaps within ~10% MAE F of the desired bandgaps as validated by density functional theory (DFT) calculations. Analysis by the principle component analysis (PCA) and modified locally linear embedding (MLLE) reveals that the generator has successfully generated structures following the statistical distribution of the real structures. It implies the possibility of the RCGAN in recognizing physical rules hidden in the high-dimensional data. The novel strategy for designing regressional GAN architecture together with the successful application to inverse design of materials would inspire further exploration in research fields beyond materials.

36 MATERIALS SCIENCE↗

Wind turbine blade design with airfoil shape control using invertible neural networks

Wind turbine blade design is a highly multidisciplinary process that involves aerodynamics, structures, controls, manufacturing, costs, and other considerations. More efficient blade designs can be found by controlling the airfoil cross-sectional shapes simultaneously with the bulk blade twist and chord distributions. Prior work has focused on incorporating panel-based aerodynamic solvers with a blade design framework to allow for airfoil shape control within the design loop in a tractable manner. Including higher fidelity aerodynamic solvers, such as computational fluid dynamics, makes the design problem computationally intractable. In this work, we couple an invertible neural network trained on high-fidelity airfoil aerodynamic data to a turbine design framework to enable the design of airfoil cross sections within a larger blade design problem. We detail the methodology of this coupled framework and showcase its efficacy by aerostructurally redesigning the IEA 15-MW reference wind turbine blade. The coupled approach reduces the cost of energy by 0.9% compared to a more conventional design approach. This work enables the inclusion of high-fidelity aerodynamic data earlier in the design process, reducing cycle time and increasing certainty in the performance of the optimal design.

17 WIND ENERGY↗

Data-Driven Protection Software to classify fault locations by protective zone in distribution systems with high PV penetration

The software contains (a) the source codes to generate Point-on-Wave (PoW) transient data for any feeder model in Alternative Transient Program (ATP) format. Codes provide options to change different steady state settings, including the loading condition and PV capacity and transient state setting like faults type, location and initiation time (b) data post-processing source code to converted data from native format to COMTRADE, csv, HDF5 (c) Docker container to train CNN to classify fault locations by protective zone. The container takes dataset and other training parameters (sampling rate, training epochs, batch size etc) as input to train CNN. The container writes back the trained CNN model, training and testing metrics and plots to the local workstation

Ramesh, Meghana↗

Tribal Colleges and Universities/American Indian Higher Education Consortium Advanced Manufacturing Technical Assistance Project

The National Nuclear Security Agency (NNSA) created a Minority Serving Institution Partnership Plan (MSIPP) to 1) align investments in a university capacity and workforce development with the NNSA mission to develop the needed skills and talent for NNSA's enduring technical workforce at the laboratories and production plants and 2) to enhance research and education at underrepresented colleges and universities. Out of this effort, MSIPP launched a new program in early FY17 focused on Tribal Colleges and Universities (TCUs). The following report summarizes the project focus and status update during this reporting period. The overall goal of this project is to establish a network of TCUs with essential advanced manufacturing facilities, associated training and education programs, and private sector and federal agency partnerships to both prepare an American Indian advanced manufacturing workforce and create economic and employment opportunities within Tribal communities through design, manufacturing, and marketing of high quality products. Some examples of high quality products involve next generation grid components such as mechanical energy storage, cabling for distribution of energy, and electrochemical energy storage enclosures.

25 ENERGY STORAGE↗

Lessons Learned From the Cesium Pond Soil Shipping Campaign at Whiteshell Laboratories - 18206

Whiteshell Laboratories (WL), located in Manitoba, Canada is the first major decommissioning project of a nuclear research establishment in Canada. With the focus now active remediation to return the site to green state, the majority of legacy radioactive waste and waste generated from decommissioning and demolition of nuclear facilities must be transported off site to Chalk River Laboratories (CRL), located in Ontario, Canada. No large radioactive waste shipment campaigns have been undertaken by WL until now. A newly established Waste Management Program specifically designed to support the WL Closure Project has executed its first major shipping campaign of radioactive material, making approximately one hundred shipments and reducing WL's total volume of low-level radioactive waste to transport by seven percent. The shipped waste was approximately 1800 m{sup 3} (2600 metric tons) of known Cs-137 contaminated soil material generated from the excavation of an Experimental Cesium Pond Project designed to study the distribution of dose received by organisms living at the mud-water interface from Cs-137 contamination. This paper outlines the cost benefit analysis undertaken to determine the most cost effective method for transporting the radioactively contaminated soil to CRL, storage challenges experienced by both WL and CRL, the selection of the most appropriate waste package, training of shippers, and the continual optimization of increasing the number of generated waste packages and the number of waste packages shipped and accepted at CRL. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Lens Model Accuracy in the Expected LSST Lensed AGN Sample

Strong gravitational lensing of active galactic nuclei (AGN) enables measurements of cosmological parameters through time-delay cosmography (TDC). With data from the upcoming LSST survey, we anticipate using a sample of O(1000) lensed AGN for TDC. To prepare for this dataset and enable this measurement, we construct and analyze a realistic mock sample of 1300 systems drawn from the OM10 (Oguri & Marshall 2010) catalog of simulated lenses with AGN sources at $z<3.1$ in order to test a key aspect of the analysis pipeline, that of the lens modeling. We realize the lenses as power law elliptical mass distributions and simulate 5-year LSST i-band coadd images. From every image, we infer the lens mass model parameters using neural posterior estimation (NPE). Focusing on the key model parameters, $θ_E$ (the Einstein Radius) and $γ_{lens}$ (the projected mass density profile slope), with consistent mass-light ellipticity correlations in test and training data, we recover $θ_E$ with less than 1% bias per lens, 6.5% precision per lens and $γ_{lens}$ with less than 3% bias per lens, 8% precision per lens. We find that lens light subtraction prior to modeling is only useful when applied to data sampled from the training prior. If emulated deconvolution is applied to the data prior to modeling, precision improves across all parameters by a factor of 2. Finally, we combine the inferred lens mass models using Bayesian Hierarchical Inference to recover the global properties of the lens sample with less than 1% bias.

Venkatraman, Padmavathi [Illinois U., Urbana; KIPA↗

Pulsed Neutron Powder Diffraction for Materials Science

The accelerator-based neutron diffraction began in the end of 60's at Tohoku University which was succeeded by the four spallation neutron facilities with proton accelerators at the High Energy Accelerator Research Organization (Japan), Argonne National Laboratory and Los Alamos Laboratory (USA), and Rutherford Appleton Laboratory (UK). Since then, the next generation source has been pursued for 20 years, and 1MW-class spallation neutron sources will be appeared in about three years at the three parts of the world: Japan, UK and USA. The joint proton accelerator project (J-PARC), a collaborative project between KEK and JAEA, is one of them. The aim of the talk is to describe about J-PARC and the neutron diffractometers being installed at the materials and life science facility of J-PARC. The materials and life science facility of J-PARC has 23 neutron beam ports and will start delivering the first neutron beam of 25 Hz from 2008 May. Until now, more than 20 proposals have been reviewed by the review committee, and accepted proposal groups have started to get fund. Those proposals include five polycrystalline diffractometers: a super high resolution powder diffractometer (SHRPD), a 0.2%-resolution powder diffractometer of Ibaraki prefecture (IPD), an engineering diffractometers (Takumi), a high intensity S(Q) diffractometer (VSD), and a high-pressure dedicated diffractometer. SHRPD, Takumi and IPD are being designed and constructed by the joint team of KEK, JAEA and Ibaraki University, whose member are originally from the KEK powder group. These three instruments are expected to start in 2008. VSD is a super high intensity diffractometer with the highest resolution of {delta}d/d = 0.3%. VSD can measure rapid time-dependent phenomena of crystalline materials as well as glass, liquid and amorphous materials. The pair distribution function will be routinely obtained by the Fourier transiformation of S(Q) data. Q range of VSD will be as wide as 0.01 A{sup -1}<Q<100 A{sup -1}. IPD is fully funded by Ibaraki prefecture for the promotion of new industries based on advanced science and technologies. It is for the first time in neutron facilities in Japan that a prefecture owns neutron instruments as well as neutron beam will be provided widely to industrial users. To make it successful, the user system is quite important because those users are expected to use IPD like chemical analyzers in their materials development process. Based on questionnaire data to several hundreds industries, IPD is designed as a versatile diffractometer including texture measurement, small angle scattering and total scattering as well as usual powder diffraction. IPD covers d range 0.15<d(A)<4 with {delta}d/d 0.15%, and covers 4<d(A)<60 with gradually changing resolution. Q range of IPD will be as wide as 0.01 A{sup -1}<Q<50A{sup -1} to be utilized for varieties of structures: local structure, nano structure and crystal structure analyses. Typical measuring time for the typical 'Rietveld-quality' data is several minutes with the sample size of laboratory X-ray: 0.4 cc. SHRPD is designed to be the world highest resolution with {delta}d/d = 0.03% without sacrificing intensity. The combination of the high quality data from HRPD and their high-precision analysis gives us information on tiny structural changes which have been overlooked. After careful examination with the moderator group five years ago, we have decided to develop a high-resolution and good S/N moderator to achieve the 0.03% resolution within 100 m flight path. This development was almost successful up to now. Instrumental simulation and radiation analysis were almost completed. The d range 0.5<d(A)<4 with {delta}d/d = 0.03%, and covers 4<d(A)<45 with gradually changing resolution. Takumi is the first priority instrument in JAEA for stress mapping inside structure materials with the highest resolution of {delta}d/d = 0.2% (corresponding to 10{approx}5 to 10{approx}6 strain precision). The typical gauge volume will be 1 mm3. JED has transmission radiography detectors to support stress mapping. Software group is planning so that basic software to cover data acquisition and data treatment should be common. Since 1 Gbyte data are typically obtained for single experiment in an instrument, the basic software is quite important. International TV conference between ISIS, IPNS, SNS has been held every month to exchange information on each development. KEK developed manyo-lib to help basic analysis. Analysis software development including powder diffraction is strongly related with the activity of the software group. However, users of IPD will be from various field of science and their background is different. It should cover wide topics and help both beginners and well-trained users. We have started with neutron intensity database, peak-search software, peak-match software, pattern simulation, whole pattern fitting, PDF and RDF analysis, and now start coding Rietveld software.

36 MATERIALS SCIENCE↗

Cybersecurity Standards for Distributed Energy Resources: Gaps and Harmonization Strategy

This report examines cybersecurity standards for Distributed Energy Resources (DERs) in light of their rapid growth and increasing integration into energy systems. It identifies critical gaps in existing frameworks, including inadequate coverage of DER-specific challenges, complexities in implementing comprehensive standards, integration issues with legacy systems, adoption hurdles for newer standards, and a lack of harmonization across regulatory landscapes. The analysis highlights vulnerabilities such as data integrity risks, unauthorized device control, and denial-of-service attacks across various DER technologies like solar PV, wind turbines, energy storage systems, and hydrogen fuel cells. The report proposes a harmonization strategy to address these deficiencies by developing unified cybersecurity requirements, certification programs, and training resources while fostering collaboration among stakeholders such as government agencies, industry groups, DER operators, manufacturers, and research institutions. A phased roadmap is outlined to refine and implement these measures through pilot testing and widespread adoption. Ultimately, the report underscores the urgent need for coordinated efforts to enhance DER cybersecurity and ensure the reliable operation of future energy systems.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Integrating very-high-resolution imagery, Sentinel-2 time-series data, and machine learning to map shrub fractional abundance across arid and semi-arid ecosystems in China

Shrub fractional abundance (SFA), the proportion of shrub cover per unit area, serves as a critical indicator of environmental aridity and ecosystem health in arid and semi-arid regions, particularly across the Mongolian steppe. However, large-scale SFA mapping in Mongolian steppe ecosystems remains challenging due to the small crown size of shrubs, their sparse distribution, and spectral overlap with coexisting low vegetation (e.g., grasses and herbs), which hinders accurate detection using coarser-resolution satellite data or traditional field surveys. To address these challenges, we developed a two-step approach that integrates very-high-resolution (VHR) imagery, time-series Sentinel-2 data, and deep learning techniques. First, we generated high-accuracy benchmark maps of individual shrub crowns from 0.5 m VHR imagery by combining manual segmentation with a hybrid deep learning framework (Dino V2 and convolutional neural networks). Second, we used these shrub crown maps as training data to build an XGBoost model for predicting SFA from 20 m Sentinel-2 time-series data, leveraging phenological information to improve estimation. We validated our approach across 70 sites (1km 2 each) in the Inner Mongolia Autonomous Region, which is representative of Mongolian steppe ecosystems. From VHR imagery, we mapped 1.31 million shrub crowns with an accuracy of R 2 = 0.92. Scaling up with Sentinel-2 data yielded regional SFA maps with an R 2 = 0.60. Further SHAP (SHapley Additive exPlanations) analysis on the developed XGBoost model revealed that phenological metrics (particularly observations in early-May, mid-July, and late-September), which distinguish shrub phenology from that of other land cover types (e.g., grasses and bare soil), were the most influential predictors of SFA. Finally, our regional SFA maps uncovered unimodal relationships between shrub distribution and climate variables, peaking at mean annual minimum temperatures near 0 °C and annual precipitation around 200 mm. Collectively, these findings demonstrate how the integration of multi-source remote sensing and machine learning can overcome historical limitations in SFA mapping, enabling accurate, spatially continuous assessments across vast Inner-Mongolian steppe ecosystems. Our framework has the potential to be applied to other steppe ecosystems and dryland ecosystems across the Mongolian steppe and beyond, offering a foundation for improved monitoring and ecological impact assessments in the face of global climate changes.

Arid and semi-arid landscapes↗

Machine Learning-Driven Solvent Screening for Biobased 2,3-Butanediol Extraction

Biobased 2,3-butanediol (2,3-BDO) is a valuable biomass-derived chemical due to its versatility in being transformed into a wide variety of products. However, the separation and purification of 2,3-BDO from fermentation broth remain a significant challenge owing to its high boiling point and hydrophilic nature. Herein, we developed a machine learning (ML)-based screening workflow that uses molecular calculations as training data and requires only a small number of experimental measurements for validation to identify alternative solvent candidates for the liquid–liquid extraction (LLE) of 2,3-BDO from aqueous solution. In particular, 130 density functional theory (DFT) calculations with the implicit solvation method not only built a correlation between the computational partition coefficient and the experimental distribution coefficient of 2,3-BDO but also parameterized an Extra-Trees ML model to screen the distribution coefficient for a wider range of 6717 organic solvents. The experimental measurements of only 24 solvents were needed to validate the computational results. A list of 50 prioritized solvents was proposed for 2,3-BDO LLE, and seven additional experimental measurements were conducted to further verify our selected solvents. The impact of the extraction temperature and solvent-to-feed ratio was also investigated for selected solvents in experiments. Furthermore, this work suggested alternative solvents for 2,3-BDO LLE and proposed a versatile workflow that requires fewer experiments and can be applied to a broader range of LLE studies.

Extraction↗

QSO photometric redshifts from SDSS, WISE, and GALEX colours

ABSTRACT Machine learning techniques, specifically the k-nearest neighbour algorithm applied to optical band colours, have had some success in predicting photometric redshifts of quasi-stellar objects (QSOs): Although the mean of differences between the spectroscopic and photometric redshifts, Δ$z$, is close to zero, the distribution of these differences remains wide and distinctly non-Gaussian. As per our previous empirical estimate of photometric redshifts, we find that the predictions can be significantly improved by adding colours from other wavebands, namely the near-infrared and ultraviolet. Self-testing this, by using half of the 33 643 strong QSO sample to train the algorithm, results in a significantly narrower spread in Δ$z$ for the remaining half of the sample. Using the whole QSO sample to train the algorithm, the same set of magnitudes return a similar spread in Δ$z$ for a sample of radio sources (quasars). Although the matching coincidence is relatively low (739 of the 3663 sources having photometry in the relevant bands), this is still significantly larger than from the empirical method (2 per cent) and thus may provide a method with which to obtain redshifts for the vast number of continuum radio sources expected to be detected with the next generation of large radio telescopes.

Curran, S. J.↗

SU-E-T-775: Use of Electronic Portal Imaging Device (EPID) for Quality Assurance (QA) of Electron Beams On Varian Truebeam System

Purpose: In a previous study we have demonstrated the feasibility of using EPID to QA electron beam parameters on a single Varian TrueBeam LINAC. This study aims to provide further investigation on (1) reproducibility of using EPID to detect electron beam energy changes on multiple machines and (2) evaluation of appropriate calibration methods to compare results from different EPIDs. Methods: Ad-hoc mode electron beam images were acquired in developer mode with XML code. Electron beam data were collected on a total of six machines from four institutions. A custom-designed double-wedge phantom was placed on the EPID detector. Two calibration methods - Pixel Sensitivity Map (PSM) and Large Source-to-Imager Distance Flood Field (LSID-FF) - were used. To test the sensitivity of EPID in detecting energy drifts, Bending Magnet Current (BMC) was detuned to invoke energy changes corresponding to ∼±1.5 mm change in R50% of PDD on two machines from two institutions. Percent depth ionization (PDI) curves were then analyzed and compared with the respective baseline images using LSID-FF calibration. For reproducibility testing, open field EPID images and images with a standard testing phantom were collected on multiple machines. Images with and without PSM correction for same energies on different machines were overlaid and compared. Results: Two pixel shifts were observed in PDI curve when energy changes exceeded the TG142 tolerance. PSM showed the potential to correct the differences in pixel response of different imagers. With PSM correction, the histogram of images differences obtained from different machines showed narrower distributions than those images without PSM correction. Conclusion: EPID is sensitive for electron energy changes and the results are reproducible on different machines. When overlaying images from different machines, PSM showed the ability to partially eliminate the intrinsic variation of various imagers. Research Funding from Varian Medical Systems Inc.Dr. Sasa Mutic receives compensation for providing patient safety training services from Varian Medical Systems, the sponsor of this study.

60 APPLIED LIFE SCIENCES↗

Humming Trains in Seismology: An Opportune Source for Probing the Shallow Crust

Abstract Seismologists are eagerly seeking new and preferably low-cost ways to map and track changes in the complex structure of the top few kilometers of the crust. By understanding it better, they can build on what is known regarding important, practical issues. These include telling us whether imminent earthquakes and volcanic eruptions are generating telltale underground signs of hazard, about mitigation of induced seismicity such as from deep injection of wastewater, how the Earth and its atmosphere couple, and where accessible natural resources are. Passive seismic imaging usually relies on blind correlations within extended recordings of Earth’s ceaseless “hum” or coda of well-mixed, small vibrations. In this article, we propose a complementary approach. It is seismic interferometry using opportune sources—specifically ones not stationary in time and moving in a well-understood configuration. Its interpretation relies on an accurate understanding of how these sources radiate seismic waves, precise timing, careful placement of pairs of listening stations, and seismic phase differentiation (surface and body waves). Massive freight trains were only recently recognized as such a persistent, powerful cultural (human activity-caused) seismic source. One train passage may generate a tremor with an energy output of a magnitude 1 earthquake and be detectable for up to 100 km from the track. We discuss the source mechanisms of train tremors and review the basic theory on sources. Finally, we present case studies of body- and surface-wave retrieval as an aid to mineral exploration in Canada and to monitoring of a southern California fault zone. We believe noise recovery from this new signal source, together with dense data acquisition technologies such as nodes or distributed acoustic sensing, will deeply transform our ability to monitor activity in the shallow crust at sharpened resolution in time and space.

Geochemistry & Geophysics↗

The Resilient Energy Platform

The provision of reliable, secure, and affordable electricity is essential to power economic growth and development. The power system is at risk from an array of natural, human-caused, and technological threats, which can cause everything from power interruption to chronic undersupply of energy. It is critical for policymakers, planners, and system operators to safeguard their power systems from these threats by proactively planning for future needs and investing in resilient power systems. Developed through the USAID-NREL Partnership, the Resilient Energy Platform is a website that provides expertly curated resources, training materials, data, tools, and direct technical assistance in planning resilient, sustainable, and secure power systems.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Deep Learning for Automated Identification of Eels in Sonar Data

Freshwater eels, such as the American eel (Anguilla rostrata) present numerous challenges related to safe downstream fish passage at hydroelectric facilities. One of those challenges is effective monitoring of their abundance, movements, and behavior to facilitate design and operation of eel protection and passage facilities. A previous EPRI study documented the ability of human analysts to reliably identify American eels in data obtained with a 1100/1800 kHz, multibeam sonar. This report describes a project to develop deep learning (a subset of artificial intelligence) tools to automate the time-consuming, subjective process of eel identification in multibeam sonar data. The project exploited new data collected in the laboratory and the existing data from the prior EPRI field study to develop and test deep learning and other data analytic tools, including wavelet filtering, differencing for static object removal, and convolutional neural network analysis. The analysis of the laboratory data demonstrated feasibility of the approach, revealed object characteristics observed with the sonar that distinguish eels from similarly sized and shaped acoustic targets, and provided additional data for algorithm selection and training. Deep learning algorithms trained and tested on the laboratory data alone achieved accuracy rates of greater than 98% when classifying acoustic images of eels and similar-sized neutrally buoyant sticks. The algorithm trained and tested on the pre-existing field data alone, and yielded classification accuracy of 9.3% false positives and 13.3% false negatives when distinguishing between eels and sticks/PVC pipes based on video clips (i.e., multiple, consecutive images). This performance is comparable to the classification accuracy achieved by human analysts in the prior study. The deep learning algorithm trained on a combination of video clips obtained in the laboratory and the field and tested on video clips from the field, was able to distinguish eels from sticks and PVC pipes (a river debris analog) of similar size with 100% accuracy. Outreach to the hardware, software, and end-user communities early in the project helped to identify needs and specify the application space. Outreach to those communities at the end of the project communicated project results and opportunities for further development. The project achieved proof of concept for automated identification of eel in multibeam sonar data. Future work should focus on acquisition of additional data for more robust algorithm training and testing; modification of the software tools to accommodate multiple acoustic targets in the acoustic field at a given time; identification of additional object classes; incorporation of motion in the object identification and classification algorithms; operationalizing the software tools, including integration with other existing sonar data analysis tools; and partnering with hardware and software providers for distribution of the software tools with their commercial products.

13 HYDRO ENERGY↗

Grid-Interactive Building Control Via Reinforcement Learning

In this work, we present the proposed two-stage reinforcement learning approach for training building controllers to help buildings to participate in DR events. The original conference paper for this research idea can be found at https://www.nrel.gov/docs/fy21osti/78000.pdf.

building control↗

Correlated Trajectory Uncertainty for Adaptive Sequential Decision Making

One of the great challenges with decision making tasks on real world systems is the fact that data is sparse and acquiring additional data is expensive. In these cases, it is often crucial to make a model of the environment to assist in making decisions. At the same time, limited data means that learned models are erroneous, making it just as important to equip the model with good predictive uncertainties. In the context of learning sequential decision making policies, these uncertainties can prove useful for informing which data to collect for the greatest improvement in policy performance \citep{mehta2021experimental, mehta2022exploration} or informing the policy about unsure regions of state and action space to avoid during test time \citep{yu2020mopo}. Additionally, assuming that realistic samples of the environment can be drawn, an adaptable policy can be trained that attempts to make optimal decisions for any given possible instance of the environment \citep{ghosh2022offline, chen2021offline}. In this work, we examine the so-called ``probabilistic neural network'' (PNN) model that is ubiquitous in model-based reinforcement learning (MBRL) works. We argue that while PNN models may have good marginal uncertainties, they form a distribution of non-smooth transition functions. Not only are these samples unrealistic and may hamper adaptability, but we also assert that this leads to poor uncertainty estimates when predicting multiple step trajectory estimates. To address this issue, we propose a simple sampling method that can be implemented on top of pre-existing models.We evaluate our sampling technique on a number of environments, including a realistic nuclear fusion task, and find that, not only do smooth transition function samples produce more calibrated uncertainties, but they also lead to better downstream performance for an adaptive policy.

Offline Reinforcement Learning↗