Adapting quantile mapping to bias correct solar radiation data
Not Available
Bailey, Maggie D. (ORCID:0000000347017849)↗
SEARCH · Search NASA
Results for “Bias Correction”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
The National Climate Data Base (NCDB): A Bias-Corrected High-Resolution Climate Dataset
Assessing renewable energy resources under future climate scenarios has been highlighted in recent years to analyze and understand potential impacts of future change in renewable generation on the power sector. Solar energy is well-known as the most plentiful among various renewable resources and usually converted to electricity using photovoltaics (PV) technologies, and the global deployment of PV technology has increased rapidly in recent decades. In this study, we develop a statistical technique to downscale the future projection of solar irradiance for PV energy-related applications. A set of Regional Climate Model (RCM)-based projections obtained from the North American Coordinated Regional Climate Downscaling Experiment (NA-CORDEX) are used as inputs to statistical methods to generate high-resolution global horizontal irradiance (GHI) over the contiguous United States (CONUS). The main steps of the statistical downscaling method include (1) regridding RCM output (0.22 degree and daily resolutions) to handle the modeled-observed data sets on a common grid, (2) correcting bias of RCM GHI using satellite-derived observation, and (3) implementing temporal and spatial downscaling to generate GHI at 8-km and hourly resolution. Basically, complex physical processes and interactions between solar radiation and various atmospheric constituents lead solar irradiance to be highly variable and uncertain. Underrepresentation of clouds from the RCM parameterizations is the main source of error and uncertainty in modeling solar irradiance. Thus, we adapt and use the high-quality satellite-derived data from the National Solar Radiation Database (NSRDB) to analyze the bias and error of RCM GHI as well as estimate the statistical parameters for spatial and temporal downscaling. This presentation will summarize the comprehensive analysis conducted to produce and assess the results under two climate scenarios (RCP4.5 and RCP8.5). We will also present a detailed validation demonstrating the strengths of the proposed downscaling method and future extension of this research.
climate data↗
Evaluating Probabilistic Deep Learning Methods for Uncertainty Quantification of Precipitation Bias Correction
Climate models often exhibit biases in their precipitation predictions, particularly underestimating high-intensity events and overestimating low precipitation. Deep learning approaches offer promising solutions, but their epistemic uncertainty associated with a deep learning–based bias correction method has not previously been quantified for reliable downstream climate impact studies. While methods for capturing the epistemic uncertainty in deep learning frameworks exist, there is currently no consensus on the best method. In this work, we compare three uncertainty quantification (UQ) methods—Deep Ensembles (DEns), Monte Carlo Dropout (MCD), and Flipout—by assessing the reliability of their uncertainty estimates using standard measures such as sharpness and calibration. These UQ methods are applied to an existing deep learning precipitation bias correction model known as UFNet: a coupled U-Net and fully connected neural network. The methods utilized to assess the models’ uncertainties are 1) calibration, which ensures that the expected probabilities of the model align with reality and 2) sharpness, which is a measure of the precision of the model’s probabilistic predictions. Of the three UQ methods evaluated, the DEns and MCD methods demonstrated the best-calibrated performance (expected calibration error of 0.36 and 0.35, respectively), compared to Flipout (0.58). In contrast, Flipout had the sharpest predictions and the highest metric performance in bias correcting precipitation—especially for higher-order moments such as kurtosis with a spatial correlation of 72% compared to 32% and 55% spatial correlation for DEns and MCD, respectively. Of the three UQ methods, MCD was found to be the most suitable method for UQ purposes based on its calibration, sharpness, and computational requirements.
Bayesian methods↗
Deep Learning-based Non-Stationary Bias Correction (NSBC)
This work develops the NSBC (non-stationary bias correction) methodology to correct temperature projection bias from E3SM. The NSBC deep learning framework consists of a three-part architecture: an auto-encoder for compressing the spatial information, an LSTM for predicting annual temperature mean, and a U-Net for capturing the residual bias in temperature. The non-stationary bias correction (NSBC) framework can correct the non-stationarity of the biases of the climate models, which significantly improves the accuracy of future temperature prediction and improves the overestimation of extreme high temperatures that many existing bias correction methods suffer from. Getting started 1. Obtain the historical climate simulation and observation data. The E3SM simulation data are available through https://aims2.llnl.gov/search/cmip6/. The pseudo observations, the Geophysical Fluid Dynamics Laboratory (GFDL)-ESM4 model (Krasting et al., 2018) are available through https://aims2.llnl.gov/search/cmip6/. The spatial resolution of E3SM and pseudo observation datasets are both regridded to a common 1° resolution grid using conservative interpolation. The regridded E3SM and pseudo observation with 1° resolution can be found throught ./data/. 2. Train the Auto-encoder model. Python 0-autoencoder.py 3. Train the LSTM Python 1-LSTM.py 4. Generate the annual mean temperature based on trained LSTM Python 2-generate_annual_mean_LSTM.py 5. Train the U-Net. Python 3-unet.py 6. Evaluation and compared with the baseline Python 4_evaluation.py Is there a deadline approaching that requires the release of yo
Lucas, Donald↗
Bias Correction in Urban Building Energy Modeling for Chicago Using Machine Learning
Urban-scale building energy modeling (UBEM) holds promise for optimizing energy usage across extensive geographic regions. However, there is a recognized bias between simulated energy consumption and actual measured data. This study, based on building data from Chicago, delved into bias correction techniques for enhancing the accuracy of UBEM energy consumption estimates. Initially, the AutoBEM simulation yielded a normalized mean bias error (NMBE) of 1.1% and 51% of Coefficient of the Variation of the Root Mean Square Error (CVRMSE) after outlier exclusion. To address this, three bias correction methods were deployed: Average Mean Bias Error based bias correction, Quantile mapping bias correction, and Machine learning-based bias correction using Linear Regression and Random Forest models. Post-correction results exhibited marked improvement. The NMBE values were diminished to 0 for Average MBE-based, 0.36 for Quantile Mapping, and 0 for Machine Learning-based corrections. Concurrently, the CVRMSE values registered reductions from an original 51 to 50.8 for Quantile Mapping, and 38.56 for Machine Learning-based corrections, pointing towards the effectiveness of specific bias correction methods in refining the precision of UBEM energy predictions. Such accurate estimations are paramount for informed energy planning and urban policy-making.
Chowdhury, Shovan↗
Bias correcting regional scale Earth system model projections: novel approach using empirical mode decomposition
Bias correction is a crucial step in using Earth system model outputs for assessments, as it adjusts systematic errors by comparing the model to observations. However, standard methods – ranging from mean-based linear scaling to distribution-based quantile mapping typically treat bias correction as a single-scale process, overlooking the fact that biases can manifest differently across daily, seasonal, and annual timescales. In this study, we propose a novel, timescale-aware bias-correction approach built on Empirical Mode Decomposition. By decomposing the meteorological signal into multiple oscillatory components and aggregating them to represent distinct timescales, we apply targeted corrections to each component, thereby preserving both short- and long-term structure in the data. Experimental illustrations show that the timescale-aware EMDBC framework matches the performance of conventional quantile-delta mapping (QDM) at the native daily scale and achieves progressively larger bias reductions at bi-weekly, seasonal, and annual scales. As a result, the proposed approach offers a more robust path to accurate and reliable Earth system projections, strengthening their utility for resilience and adaptation planning.
Ganguli, Arkaprabha [Argonne National Laboratory (↗
Downscaling and bias-correction contribute considerable uncertainty to local climate projections in CMIP6
Abstract Efforts to diagnose the risks of a changing climate often rely on downscaled and bias-corrected climate information, making it important to understand the uncertainties and potential biases of this approach. Here, we perform a variance decomposition to partition uncertainty in global climate projections and quantify the relative importance of downscaling and bias-correction. We analyze simple climate metrics such as annual temperature and precipitation averages, as well as several indices of climate extremes. We find that downscaling and bias-correction often contribute substantial uncertainty to local decision-relevant climate outcomes, though our results are strongly heterogeneous across space, time, and climate metrics. Our results can provide guidance to impact modelers and decision-makers regarding the uncertainties associated with downscaling and bias-correction when performing local-scale analyses, as neglecting to account for these uncertainties may risk overconfidence relative to the full range of possible climate futures.
54 ENVIRONMENTAL SCIENCES↗
Global variance decomposition of downscaled and bias-corrected CMIP6 climate projections
This dataset provides the results of a global variance decomposition of downscaled and bias-corrected CMIP6 climate projections. The total projection variance for a set of climate metrics is partitioned into contributions from: scenario uncertainty, model/GCM uncertainty, downscaling and bias-correction uncertainty, and interannual variability. The contribution from each source is expressed as a percentage of the total variance. Seven climate metrics are analyzed: Annual average temperature (avg_tas.nc) Annual total precipitation (tot_pr.nc) Annual maximum of daily maximum temperature (max_tasmax.nc) Annual maximum 1-day precipitation (max_pr.nc) Annual number of extremely hot days (hot_days.nc) Annual number of extremely wet days (wet_days.nc) Annual number of dry days (dry_days.nc) Extremely hot/wet days are defined to occur when temperature/precipitation exceeds the local 99th percentile defined over 1980-2014. Dry days are defined to occur when daily precipitation is less than 1mm. all_metrics_timesliced.nc gives the results for all metrics averaged over three 20-year periods: 2020-2039, 2050-2069, 2080-2099. For more details on the methods, see: Lafferty & Sriver, Downscaling and bias-correction contribute considerable uncertainty to local climate projections in CMIP6, npj Climate & Atmospheric Science (2023) An interactive visualization of this data can be found at: https://lafferty-sriver-2023-downscaling-uncertainty.msdlive.org
Lafferty, David↗
Bias Corrected NOAA HRRR Wind Resource Data for Grid Integration Applications
To address the need for regularly updated wind resource data, NREL has processed the High-Resolution Rapid Refresh (HRRR) outputs for use in grid integration modeling. The HRRR is an hourly-updated operational forecast product produced by the National Oceanic and Atmospheric Administration (NOAA) (Dowell et al., 2022). Several barriers have prevented the HRRR's widespread proliferation in the wind energy industry: missing timesteps (prior to 2019), challenging file format for wind energy analysis, limited vertical height resolution, and negative bias versus legacy WIND Toolkit data (2007-2013). NREL has applied re-gridding, interpolation, and bias-correction to the native HRRR data to overcome these limitations. This results in the now-publicly-available bias corrected and interpolated HRRR (BC-HRRR) dataset for weather years 2015 to 2023. Bias correction is necessary for wind resource consistency across weather years to be used simultaneously in planning-focused grid integration studies alongside the original WIND Toolkit data. We show that quantile mapping with the WIND Toolkit as a historical baseline is an effective method for bias correcting the interpolated HRRR data: the BC-HRRR has reduced mean bias versus comparable gridded wind resource datasets (+0.12 m/s versus Vortex) and has very low mean bias versus ground measurement stations (+0.01 m/s) (Buster et al., 2024). BC-HRRR's consistency with the legacy WIND Toolkit allows NREL to extend grid integration analysis to 15+ weather years of wind data with low-overhead extensibility to future years as they are made available by NOAA. As with historical datasets like the WIND Toolkit, BC-HRRR is intended for use in grid integration modeling (e.g., capacity expansion, production cost, and resource adequacy modeling) both independently and alongside the legacy WIND Toolkit.
Array↗
A Machine Learning Bias Correction on Large–Scale Environment of High–Impact Weather Systems in E3SM Atmosphere Model
Large–scale dynamical and thermodynamical processes are common environmental drivers of high–impact weather systems causing extreme weather events. However, such large–scale environmental conditions often display systematic biases in climate simulations, posing challenges to evaluating high–impact weather systems and extreme weather events. In this paper, a machine learning (ML) approach was employed to bias correct the large–scale wind, temperature, and humidity simulated by the atmospheric component of the Energy Exascale Earth System Model (E3SM) at ~1° resolution. The usefulness of the ML approach for extreme weather analysis was demonstrated with a focus on three high–impact weather systems, including tropical cyclones (TCs), extratropical cyclones (ETCs), and atmospheric rivers (ARs). We show that the ML model can effectively reduce climate bias in large–scale wind, temperature, and humidity while preserving their responses to imposed climate change perturbations. The bias correction is found to directly improve water vapor transport associated with ARs, and representations of thermodynamical flows associated with ETCs. When the bias–corrected large–scale winds are used to drive a synthetic TC track forecast model over the Atlantic basin, the resulting TC track density agrees better with that of the TC track model driven by observed winds. In addition, the ML model insignificantly interferes with the mean climate change signals of large–scale storm environments as well as the occurrence and intensity of three weather systems. This study suggests that the proposed ML approach can be used to improve the downscaling of extreme weather events by providing more realistic large–scale storm environments simulated by low–resolution climate models.
54 ENVIRONMENTAL SCIENCES↗
Future Climate Projections for South Florida: Improving the Accuracy of Air Temperature and Precipitation Extremes With a Hybrid Statistical Bias Correction Technique
Projecting future climate variables is essential for comprehending the potential impacts on hydroclimatic hazards like floods and droughts. Evaluating these impacts is challenging due to the coarse spatial resolution of global climate models (GCMs); therefore, bias correction is widely used. Here, we applied two statistical methods—standard empirical quantile mapping (EQM) and a hybrid approach, EQM with linear correction (EQM-LIN)—to bias correct precipitation and air temperature simulated by nine GCMs. We used historical observations from 20 weather stations across South Florida to project future climate under three shared socioeconomic pathways (SSPs). Compared to the EQM, the hybrid EQM-LIN method improved R 2 of daily quantiles by up to 30% over the historical period and improved MAE up to 70% in months that contain most extreme values. Projected extreme precipitation at the weather stations showed that, compared to the EQM-LIN, the EQM method underestimates the high quantiles by up to 26% in SSP585. The projected changes in annual maximum precipitation from historical period (1985–2014) to near future (2040–2069) and far future (2070–2100) were between 2% and 16% across the study area. Projected future precipitation suggested a slight decrease during summer but an increase in fall. This, along with rising summer temperatures, suggested that South Florida can experience rapid oscillations from warmer summers and increased flooding in fall under future climate. Additionally, our comparative analyses with globally and nationally downscaled studies showed that such coarse scale studies do not represent the climatic extremes well, particularly for high quantile precipitation.
54 ENVIRONMENTAL SCIENCES↗
Analytical noise bias correction for precise weak lensing shear inference
Noise bias is a significant source of systematic error in weak gravitational lensing measurements that must be corrected to satisfy the stringent standards of modern imaging surveys in the era of precision cosmology. This paper reviews the analytical noise bias correction method and provides analytical derivations demonstrating that we can recover shear to its second order using the ‘renoising’ noise bias correction approach introduced by METACALIBRATION. We implement this analytical noise bias correction within the AnaCal shear estimation framework and propose several enhancements to the noise bias correction algorithm. We evaluate the improved AnaCal using simulations designed to replicate Rubin Legacy Survey of Space and Time (LSST) imaging data. These simulations feature semi-realistic galaxies and stars, complete with representative distributions of magnitudes and Galactic spatial density. We conduct tests under various observational challenges, including cosmic rays, defective CCD columns, bright star saturation, bleed trails, and spatially variable point spread functions. Our results indicate a multiplicative bias in weak lensing shear recovery of less than a few tenths of a per cent, meeting LSST Dark Energy Science Collaboration requirements without requiring calibration from external image simulations. Additionally, our algorithm achieves rapid processing, handling one galaxy in less than a millisecond.
79 ASTRONOMY AND ASTROPHYSICS↗
Bias Correcting NOAA's High-Resolution Rapid Refresh (HRRR) Wind Resource Data for Grid Integration Applications [Slides]
Many weather years of high-quality wind data are widely accepted in the grid integration community to be important for studying wind energy technical potential, energy system operations, and grid resilience. NREL makes high-quality wind and solar resource data available. NREL's Grid-Atmosphere workshop (March 2024) identified NREL National Solar Radiation Database as widely used in grid integration modeling, but there is less agreement on commonly used wind datasets. One important factor identified by ESIG's 2023 report 'Weather Dataset Needs for Planning and Analyzing Modern Power Systems' for gold standard wind data is regular updates. To address the need for regular updates, NREL's team can now process all currently available and regularly updated High-Resolution Rapid Refresh (HRRR) outputs. HRRR is an hourly-updated operational forecast product produced by the National Oceanic and Atmospheric Administration (NOAA) (Dowell et al., 2022). One barrier to NREL using HRRR is systematic bias and consistency with NREL's existing wind datasets (e.g. WIND Toolkit, 'WTK') across weather years. To address this barrier, we show that the HRRR can be interpolated and bias-corrected to be consistent with NRE's existing datasets. We call the new dataset BC-HRRR (bias-corrected HRRR). As with historical datasets like the WTK, BC-HRRR is intended for use in grid integration modeling (e.g., capacity expansion, production cost, and resource adequacy modeling). BC-HRRR's (2015-present) consistency with WTK (2007-2013) allows NREL to extend internal grid integration tooling with 15+ weather years of wind data with low-overhead extensibility to future years as they are made available by NOAA. The rest of this slide deck documents the BC-HRRR processing methods, validation, and its implications for intended use.
17 WIND ENERGY↗
Machine learning-based bias-corrected future projections of ozone concentrations from a chemistry-climate model
Reliable projection of future near-surface ozone is crucial for air quality management and health risk assessment. However, potential biases in spatial distribution, magnitude and trends in ozone concentrations simulated by global chemistry-climate models limit their applicability in regional-scale evaluations. In this study, LightGBM, a machine learning (ML) algorithm is applied to correct biases in CESM2-simulated ozone concentrations over China, the United States and Europe and calibrate future ozone projections under two diverse Shared Socioeconomic Pathways (SSP1-2.6 and SSP5-8.5) scenarios from 2020 to 2060. The ML-based correction significantly improves the spatial distribution and reduces the model bias by 40%–60%. It also reverses the potentially incorrect trend of ozone change under SSP1-2.6 in eastern China. When applying ML-based bias correction to CESM2 future projections, warm season mean ozone concentrations decrease across China, the United States, and Europe by –13.5, –17.9, and –13.7 µg/m³, respectively, between 2020 and 2060 in SSP1-2.6, while they increase by 9.4, 2.0, and 5.2 µg/m³ in SSP5-8.5. Decomposition analysis show that changes in anthropogenic emissions dominate future ozone changes in both scenarios, while strong climate penalty from ozone changes occurs in polluted eastern China and climate benefit is found in western China, the United States and Europe under SSP5-8.5. These findings demonstrate the value of combining ML with chemistry-climate models to produce more accurate air quality projections, thereby informing more effective and region-specific environmental protection strategies.
Chemistry Model↗
UFNet: Joint U-Net and Fully Connected Neural Network to Bias Correct Precipitation Predictions from
Paper information. Shuang Yu, Indrasis Chakraborty, Gemma J. Anderson, Donald D. Lucas, Yannic Lops, and Daniel Galea. UFNet: Joint U-Net and fully connected neural network to bias correct precipitation predictions from climate models. Artificial Intelligence for the Earth Systems, 2024. Overview. This work develops the UFNet methodology to correct E3SM historical precipitation projection bias. The UFNet deep learning framework consists of a two-part architecture: a U-Net convolutional network to capture the spatiotemporal distribution of precipitation and a fully connected network to capture the distribution of higher-order statistics. The joint network, termed UFNet, can simultaneously improve the spatial structure of the modeled precipitation and capture the distribution of extreme precipitation values. Below we provide guidance for applying UFNet to correct the Energy Exascale Earth System Model (E3SM; Golaz et al. 2019) daily precipitation projection over the contiguous United States (CONUS). Getting started 1. Obtain the historical climate simulation and observation data. The E3SM historical simulation data are available through https://aims2.llnl.gov/search/cmip6/. The CPC unified gauge-based analysis of daily precipitation can be found through https://psl.noaa.gov/data/gridded/data.cpc.globalprecip.html. The ECMWF atmospheric reanalysis of the 20th century (ERA-20C) data are available through https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era-20c. The spatial resolution of E3SM and observed datasets are both regridded to a common 1° resolution grid using conservative interpolation. The regridded E3SM, CPC and ERA-20C with 1° resolution can be found throught ./data/. 2. Train the fully connected network (DNN) Python train_dnn.py 3. Train the UFNet Python train_ufnet.py 4. Evaluation and compared with the baseline Python evaluation.py
Lucas, DonaldD↗
Bias Correction and Statistical Downscaling of Future Solar Irradiance Projections Using the NSRDB
Assessing renewable energy resources under future climate scenarios has been highlighted to understand potential impacts of future climate change in renewable generation on the power sector. Climate model projection has been recognized by the renewable energy community as a useful data set to analyze the impacts of future climate change on renewable resources. However, future climate projections generated from general circulation models (GCMs) contain inherent biases that need to be corrected for accurate analysis of future projections of climate variables. In addition, the coarse spatiotemporal resolution of GCMs needs to be improved for regional climate studies. In this work, we develop statistical methods to downscale future projections of global horizontal irradiance (GHI) in a computationally efficient way. Our approach builds statistical downscaling models that correct bias of climate projection of GHI and downscale the future GHI projection from daily-scale to hourly-scale. The National Solar Radiation Database (NSRDB) is used to calibrate the statistical models and validate the downscaled GHI projections across the contiguous United State (CONUS). Preliminary results show that the statistical approach efficiently downscales climate projections of GHI with a nBIAS of 3%, nMAE of 34 % and nRMSE of 46% calculated against NSRDB for CONUS. This study describes the implemented methodology and initial results as well as future research to create high-resolution climate data sets for solar energy applications.
analytical models↗
Is Bias Correction in Dynamical Downscaling Defensible?
Localized projections of 21st‐century hydroclimate variables obtained from downscaling Global Climate Model (GCM) output are central to informing regional impact assessments and infrastructure planning. Regional GCM biases can be significant and, for dynamical downscaling, can be addressed either before (a priori) or after (a posteriori) downscaling. However, a priori bias correction (APBC) has generally unexplored effects on climate change signals. Here we analyze dynamically downscaled solutions of CMIP6 GCMs over the Western U.S., with and without APBC, and quantify APBC's impact on climate change signals relative to other irreducible uncertainty sources. For temperature and precipitation, the uncertainty introduced by APBC is negligible compared to that arising from GCM choice or internal variability. Furthermore, APBC greatly reduces regional models' unrealistically high snow‐water‐equivalent (SWE) biases that result directly from GCM errors. We leverage this finding to encourage the dynamical downscaling community to adopt APBC as a standard operating procedure.
Risser, Mark D.↗
Bias Correction and Statistical Downscaling of Solar Radiation Using NA-CORDEX and the NSRDB
The current state-of-art for estimating long-term PV production uses long-term estimates of solar radiation variables, such as global horizontal irradiance (GHI), from previous years. This data is used in models such as the System Advisor Model (SAM) or PYSyst to predict annual production for a PV plant. This information is then used to estimate the production over the next 20 years (a typical plant lifetime) under the assumption that the variability over the current period is representative of the future. As the PV industry moves to extend plant lifetimes to 50 years the current assumptions of representativeness of weather may not be appropriate. This is especially true as our climate changes rapidly. To assess long-term PV production, future projections for solar radiation based on projected carbon emissions are readily available in regional and global climate models. However, climate model projections contain inherent biases that may need to be corrected for accurate analysis of future projections of climate variables. Several studies have analyzed projections of solar radiation for future years, however the accuracy of the model output compared to current and historic data has not been widely studied. Chen (2021) showed that available climate models do not accurately represent solar radiation in some cases, over-projecting GHI at the surface while under-projecting its obstructions, such as clouds and aerosols. This works aims to (1) increase understanding of the accuracy of solar radiation currently available in global and regional climate models and (2) implement bias correction through linear models based on reanalysis data compared to observed solar radiation. The latter aim will be conducted using available observed solar radiation data and modeled data from several regional climate models (RCMs). The bias correction method will be applied to projections of solar radiation resulting in a more accurate representation of the future of solar production.
climate data↗