Adapting quantile mapping to bias correct solar radiation data
Not Available
Bailey, Maggie D. (ORCID:0000000347017849)↗
SEARCH · Search NASA
Results for “Bias Correction”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
The National Climate Data Base (NCDB): A Bias-Corrected High-Resolution Climate Dataset
Assessing renewable energy resources under future climate scenarios has been highlighted in recent years to analyze and understand potential impacts of future change in renewable generation on the power sector. Solar energy is well-known as the most plentiful among various renewable resources and usually converted to electricity using photovoltaics (PV) technologies, and the global deployment of PV technology has increased rapidly in recent decades. In this study, we develop a statistical technique to downscale the future projection of solar irradiance for PV energy-related applications. A set of Regional Climate Model (RCM)-based projections obtained from the North American Coordinated Regional Climate Downscaling Experiment (NA-CORDEX) are used as inputs to statistical methods to generate high-resolution global horizontal irradiance (GHI) over the contiguous United States (CONUS). The main steps of the statistical downscaling method include (1) regridding RCM output (0.22 degree and daily resolutions) to handle the modeled-observed data sets on a common grid, (2) correcting bias of RCM GHI using satellite-derived observation, and (3) implementing temporal and spatial downscaling to generate GHI at 8-km and hourly resolution. Basically, complex physical processes and interactions between solar radiation and various atmospheric constituents lead solar irradiance to be highly variable and uncertain. Underrepresentation of clouds from the RCM parameterizations is the main source of error and uncertainty in modeling solar irradiance. Thus, we adapt and use the high-quality satellite-derived data from the National Solar Radiation Database (NSRDB) to analyze the bias and error of RCM GHI as well as estimate the statistical parameters for spatial and temporal downscaling. This presentation will summarize the comprehensive analysis conducted to produce and assess the results under two climate scenarios (RCP4.5 and RCP8.5). We will also present a detailed validation demonstrating the strengths of the proposed downscaling method and future extension of this research.
climate data↗
Statistical Downscaling and Bias Correction of Climate Model Outputs for Climate Change Impact Assessment in the U.S. Northeast
Statistical downscaling can be used to efficiently downscale a large number of General Circulation Model (GCM) outputs to a fine temporal and spatial scale. To facilitate regional impact assessments, this study statistically downscales (to 1∕8deg spatial resolution) and corrects the bias of daily maximum and minimum temperature and daily precipitation data from six GCMs and four Regional Climate Models (RCMs) for the northeast United States (US) using the Statistical Downscaling and Bias Correction (SDBC) approach. Based on these downscaled data from multiple models, five extreme indices were analyzed for the future climate to quantify future changes of climate extremes. For a subset of models and indices, results based on raw and bias corrected model outputs for the present-day climate were compared with observations, which demonstrated that bias correction is important not only for GCM outputs, but also for RCM outputs. For future climate, bias correction led to a higher level of agreements among the models in predicting the magnitude and capturing the spatial pattern of the extreme climate indices. We found that the incorporation of dynamical downscaling as an intermediate step does not lead to considerable differences in the results of statistical downscaling for the study domain.
extreme climate index↗
Improving Surface PM2.5 Forecasts in the United States Using an Ensemble of Chemical Transport Model Outputs: 2. Bias Correction with Satellite Data for Rural Areas
This work serves as the second of a two-part study to improve surface PM2.5 forecasts in the continental U.S. through the integrated use of multi satellite aerosol optical depth (AOD) products (MODIS Terra/Aqua and VIIRS DT/DB), multichemical transport model (CTM) (GEOS-Chem, WRF-Chem, and CMAQ) outputs, and ground observations. In Part I of the study, an ensemble Kalman filter (KF) technique using three CTM outputs and ground observations was developed to correct forecast bias and generate a single best forecast of PM2.5 for next day over non rural areas that have surface PM2.5 measurements in the proximity of 125 km. Here, with AOD data, we extended the bias correction into rural areas where the closest air quality monitoring station is at least 125–300 km away. First, we ensembled all of satellite AOD products to yield the single best AOD. Second, we corrected daily PM2.5 in rural areas from multiple models through the AOD spatial pattern between these areas and non rural areas, referred to as “extended ground truth” or EGT, for the present day. Lastly, we applied the KF technique to reduce the forecast bias for next day using the EGT. Our results find that the ensemble of bias-corrected daily PM2.5 from three CTMs for both today and next day show the best performance. Together, the two-part study develops a multimodel and multi-AOD bias-correction technique that has the potential to improve PM2.5 forecasts in both rural and non rural areas in near real time, and be readily implemented at state levels.
Huanxin Zhang↗
Evaluating Probabilistic Deep Learning Methods for Uncertainty Quantification of Precipitation Bias Correction
Climate models often exhibit biases in their precipitation predictions, particularly underestimating high-intensity events and overestimating low precipitation. Deep learning approaches offer promising solutions, but their epistemic uncertainty associated with a deep learning–based bias correction method has not previously been quantified for reliable downstream climate impact studies. While methods for capturing the epistemic uncertainty in deep learning frameworks exist, there is currently no consensus on the best method. In this work, we compare three uncertainty quantification (UQ) methods—Deep Ensembles (DEns), Monte Carlo Dropout (MCD), and Flipout—by assessing the reliability of their uncertainty estimates using standard measures such as sharpness and calibration. These UQ methods are applied to an existing deep learning precipitation bias correction model known as UFNet: a coupled U-Net and fully connected neural network. The methods utilized to assess the models’ uncertainties are 1) calibration, which ensures that the expected probabilities of the model align with reality and 2) sharpness, which is a measure of the precision of the model’s probabilistic predictions. Of the three UQ methods evaluated, the DEns and MCD methods demonstrated the best-calibrated performance (expected calibration error of 0.36 and 0.35, respectively), compared to Flipout (0.58). In contrast, Flipout had the sharpest predictions and the highest metric performance in bias correcting precipitation—especially for higher-order moments such as kurtosis with a spatial correlation of 72% compared to 32% and 55% spatial correlation for DEns and MCD, respectively. Of the three UQ methods, MCD was found to be the most suitable method for UQ purposes based on its calibration, sharpness, and computational requirements.
Bayesian methods↗
Deep Learning-based Non-Stationary Bias Correction (NSBC)
This work develops the NSBC (non-stationary bias correction) methodology to correct temperature projection bias from E3SM. The NSBC deep learning framework consists of a three-part architecture: an auto-encoder for compressing the spatial information, an LSTM for predicting annual temperature mean, and a U-Net for capturing the residual bias in temperature. The non-stationary bias correction (NSBC) framework can correct the non-stationarity of the biases of the climate models, which significantly improves the accuracy of future temperature prediction and improves the overestimation of extreme high temperatures that many existing bias correction methods suffer from. Getting started 1. Obtain the historical climate simulation and observation data. The E3SM simulation data are available through https://aims2.llnl.gov/search/cmip6/. The pseudo observations, the Geophysical Fluid Dynamics Laboratory (GFDL)-ESM4 model (Krasting et al., 2018) are available through https://aims2.llnl.gov/search/cmip6/. The spatial resolution of E3SM and pseudo observation datasets are both regridded to a common 1° resolution grid using conservative interpolation. The regridded E3SM and pseudo observation with 1° resolution can be found throught ./data/. 2. Train the Auto-encoder model. Python 0-autoencoder.py 3. Train the LSTM Python 1-LSTM.py 4. Generate the annual mean temperature based on trained LSTM Python 2-generate_annual_mean_LSTM.py 5. Train the U-Net. Python 3-unet.py 6. Evaluation and compared with the baseline Python 4_evaluation.py Is there a deadline approaching that requires the release of yo
Lucas, Donald↗
Bias Correction in Urban Building Energy Modeling for Chicago Using Machine Learning
Urban-scale building energy modeling (UBEM) holds promise for optimizing energy usage across extensive geographic regions. However, there is a recognized bias between simulated energy consumption and actual measured data. This study, based on building data from Chicago, delved into bias correction techniques for enhancing the accuracy of UBEM energy consumption estimates. Initially, the AutoBEM simulation yielded a normalized mean bias error (NMBE) of 1.1% and 51% of Coefficient of the Variation of the Root Mean Square Error (CVRMSE) after outlier exclusion. To address this, three bias correction methods were deployed: Average Mean Bias Error based bias correction, Quantile mapping bias correction, and Machine learning-based bias correction using Linear Regression and Random Forest models. Post-correction results exhibited marked improvement. The NMBE values were diminished to 0 for Average MBE-based, 0.36 for Quantile Mapping, and 0 for Machine Learning-based corrections. Concurrently, the CVRMSE values registered reductions from an original 51 to 50.8 for Quantile Mapping, and 38.56 for Machine Learning-based corrections, pointing towards the effectiveness of specific bias correction methods in refining the precision of UBEM energy predictions. Such accurate estimations are paramount for informed energy planning and urban policy-making.
Chowdhury, Shovan↗
Bias correcting regional scale Earth system model projections: novel approach using empirical mode decomposition
Bias correction is a crucial step in using Earth system model outputs for assessments, as it adjusts systematic errors by comparing the model to observations. However, standard methods – ranging from mean-based linear scaling to distribution-based quantile mapping typically treat bias correction as a single-scale process, overlooking the fact that biases can manifest differently across daily, seasonal, and annual timescales. In this study, we propose a novel, timescale-aware bias-correction approach built on Empirical Mode Decomposition. By decomposing the meteorological signal into multiple oscillatory components and aggregating them to represent distinct timescales, we apply targeted corrections to each component, thereby preserving both short- and long-term structure in the data. Experimental illustrations show that the timescale-aware EMDBC framework matches the performance of conventional quantile-delta mapping (QDM) at the native daily scale and achieves progressively larger bias reductions at bi-weekly, seasonal, and annual scales. As a result, the proposed approach offers a more robust path to accurate and reliable Earth system projections, strengthening their utility for resilience and adaptation planning.
Ganguli, Arkaprabha [Argonne National Laboratory (↗
Downscaling and bias-correction contribute considerable uncertainty to local climate projections in CMIP6
Abstract Efforts to diagnose the risks of a changing climate often rely on downscaled and bias-corrected climate information, making it important to understand the uncertainties and potential biases of this approach. Here, we perform a variance decomposition to partition uncertainty in global climate projections and quantify the relative importance of downscaling and bias-correction. We analyze simple climate metrics such as annual temperature and precipitation averages, as well as several indices of climate extremes. We find that downscaling and bias-correction often contribute substantial uncertainty to local decision-relevant climate outcomes, though our results are strongly heterogeneous across space, time, and climate metrics. Our results can provide guidance to impact modelers and decision-makers regarding the uncertainties associated with downscaling and bias-correction when performing local-scale analyses, as neglecting to account for these uncertainties may risk overconfidence relative to the full range of possible climate futures.
54 ENVIRONMENTAL SCIENCES↗
Global variance decomposition of downscaled and bias-corrected CMIP6 climate projections
This dataset provides the results of a global variance decomposition of downscaled and bias-corrected CMIP6 climate projections. The total projection variance for a set of climate metrics is partitioned into contributions from: scenario uncertainty, model/GCM uncertainty, downscaling and bias-correction uncertainty, and interannual variability. The contribution from each source is expressed as a percentage of the total variance. Seven climate metrics are analyzed: Annual average temperature (avg_tas.nc) Annual total precipitation (tot_pr.nc) Annual maximum of daily maximum temperature (max_tasmax.nc) Annual maximum 1-day precipitation (max_pr.nc) Annual number of extremely hot days (hot_days.nc) Annual number of extremely wet days (wet_days.nc) Annual number of dry days (dry_days.nc) Extremely hot/wet days are defined to occur when temperature/precipitation exceeds the local 99th percentile defined over 1980-2014. Dry days are defined to occur when daily precipitation is less than 1mm. all_metrics_timesliced.nc gives the results for all metrics averaged over three 20-year periods: 2020-2039, 2050-2069, 2080-2099. For more details on the methods, see: Lafferty & Sriver, Downscaling and bias-correction contribute considerable uncertainty to local climate projections in CMIP6, npj Climate & Atmospheric Science (2023) An interactive visualization of this data can be found at: https://lafferty-sriver-2023-downscaling-uncertainty.msdlive.org
Lafferty, David↗
Bias Corrected NOAA HRRR Wind Resource Data for Grid Integration Applications
To address the need for regularly updated wind resource data, NREL has processed the High-Resolution Rapid Refresh (HRRR) outputs for use in grid integration modeling. The HRRR is an hourly-updated operational forecast product produced by the National Oceanic and Atmospheric Administration (NOAA) (Dowell et al., 2022). Several barriers have prevented the HRRR's widespread proliferation in the wind energy industry: missing timesteps (prior to 2019), challenging file format for wind energy analysis, limited vertical height resolution, and negative bias versus legacy WIND Toolkit data (2007-2013). NREL has applied re-gridding, interpolation, and bias-correction to the native HRRR data to overcome these limitations. This results in the now-publicly-available bias corrected and interpolated HRRR (BC-HRRR) dataset for weather years 2015 to 2023. Bias correction is necessary for wind resource consistency across weather years to be used simultaneously in planning-focused grid integration studies alongside the original WIND Toolkit data. We show that quantile mapping with the WIND Toolkit as a historical baseline is an effective method for bias correcting the interpolated HRRR data: the BC-HRRR has reduced mean bias versus comparable gridded wind resource datasets (+0.12 m/s versus Vortex) and has very low mean bias versus ground measurement stations (+0.01 m/s) (Buster et al., 2024). BC-HRRR's consistency with the legacy WIND Toolkit allows NREL to extend grid integration analysis to 15+ weather years of wind data with low-overhead extensibility to future years as they are made available by NOAA. As with historical datasets like the WIND Toolkit, BC-HRRR is intended for use in grid integration modeling (e.g., capacity expansion, production cost, and resource adequacy modeling) both independently and alongside the legacy WIND Toolkit.
Array↗
Implementation of Coupled Skin Temperature Analysis and Bias Correction in a Global Atmospheric Data Assimilation System
The NASA/NCAR Finite Volume GCM (fvGCM) with the NCAR CLM (Community Land Model) version 2.0 was integrated into the NASA/GMAO Finite Volume Data Assimilation System (fvDAS). A new method was developed for coupled skin temperature assimilation and bias correction where the analysis increment and bias correction term is passed into the CLM2 and considered a forcing term in the solution to the energy balance. For our purposes, the fvDAS CLM2 was run at 1 deg. x 1.25 deg. horizontal resolution with 55 vertical levels. We assimilate the ISCCP-DX (30 km resolution) surface temperature product. The atmospheric analysis was performed 6-hourly, while the skin temperature analysis was performed 3-hourly. The bias correction term, which was updated at the analysis times, was added to the skin temperature tendency equation at every timestep. In this presentation, we focus on the validation of the surface energy budget at the in situ reference sites for the Coordinated Enhanced Observation Period (CEOP). We will concentrate on sites that include independent skin temperature measurements and complete energy budget observations for the month of July 2001. In addition, MODIS skin temperature will be used for validation. Several assimilations were conducted and preliminary results will be presented.
Radakovich, Jon↗
A Machine Learning Bias Correction on Large–Scale Environment of High–Impact Weather Systems in E3SM Atmosphere Model
Large–scale dynamical and thermodynamical processes are common environmental drivers of high–impact weather systems causing extreme weather events. However, such large–scale environmental conditions often display systematic biases in climate simulations, posing challenges to evaluating high–impact weather systems and extreme weather events. In this paper, a machine learning (ML) approach was employed to bias correct the large–scale wind, temperature, and humidity simulated by the atmospheric component of the Energy Exascale Earth System Model (E3SM) at ~1° resolution. The usefulness of the ML approach for extreme weather analysis was demonstrated with a focus on three high–impact weather systems, including tropical cyclones (TCs), extratropical cyclones (ETCs), and atmospheric rivers (ARs). We show that the ML model can effectively reduce climate bias in large–scale wind, temperature, and humidity while preserving their responses to imposed climate change perturbations. The bias correction is found to directly improve water vapor transport associated with ARs, and representations of thermodynamical flows associated with ETCs. When the bias–corrected large–scale winds are used to drive a synthetic TC track forecast model over the Atlantic basin, the resulting TC track density agrees better with that of the TC track model driven by observed winds. In addition, the ML model insignificantly interferes with the mean climate change signals of large–scale storm environments as well as the occurrence and intensity of three weather systems. This study suggests that the proposed ML approach can be used to improve the downscaling of extreme weather events by providing more realistic large–scale storm environments simulated by low–resolution climate models.
54 ENVIRONMENTAL SCIENCES↗
Future Climate Projections for South Florida: Improving the Accuracy of Air Temperature and Precipitation Extremes With a Hybrid Statistical Bias Correction Technique
Projecting future climate variables is essential for comprehending the potential impacts on hydroclimatic hazards like floods and droughts. Evaluating these impacts is challenging due to the coarse spatial resolution of global climate models (GCMs); therefore, bias correction is widely used. Here, we applied two statistical methods—standard empirical quantile mapping (EQM) and a hybrid approach, EQM with linear correction (EQM-LIN)—to bias correct precipitation and air temperature simulated by nine GCMs. We used historical observations from 20 weather stations across South Florida to project future climate under three shared socioeconomic pathways (SSPs). Compared to the EQM, the hybrid EQM-LIN method improved R 2 of daily quantiles by up to 30% over the historical period and improved MAE up to 70% in months that contain most extreme values. Projected extreme precipitation at the weather stations showed that, compared to the EQM-LIN, the EQM method underestimates the high quantiles by up to 26% in SSP585. The projected changes in annual maximum precipitation from historical period (1985–2014) to near future (2040–2069) and far future (2070–2100) were between 2% and 16% across the study area. Projected future precipitation suggested a slight decrease during summer but an increase in fall. This, along with rising summer temperatures, suggested that South Florida can experience rapid oscillations from warmer summers and increased flooding in fall under future climate. Additionally, our comparative analyses with globally and nationally downscaled studies showed that such coarse scale studies do not represent the climatic extremes well, particularly for high quantile precipitation.
54 ENVIRONMENTAL SCIENCES↗
Downscaling Atmosphere-Ocean Global Climate Model Precipitation Simulations over Africa Using Bias-Corrected Lateral and Lower Boundary Conditions
A prequel study showed that dynamic downscaling using a regional climate model (RCM) over Africa improved the Goddard Institute for Space Studies Atmosphere-Ocean Global Climate Model (GISS AOGCM: ModelE) simulation of June–September rainfall patterns over Africa. The current study applies bias corrections to the lateral and lower boundary data from the AOGCM driving the RCM, based on the comparison of a 30-year simulation to the actual climate. The analysis examines the horizontal pattern of June–September total accumulated precipitation, the time versus latitude evolution of zonal mean West Africa (WA) precipitation (showing monsoon onset timing), and the latitude versus altitude cross-section of zonal winds over WA (showing the African Easterly Jet and the Tropical Easterly Jet). The study shows that correcting for excessively warm AOGCM Atlantic sea-surface temperatures (SSTs) improves the simulation of key features, whereas applying 30-year mean bias corrections to atmospheric variables driving the RCM at the lateral boundaries does not improve the RCM simulations. We suggest that AOGCM climate projections for Africa should benefit from downscaling by nesting an RCM that has demonstrated skill in simulating African climate, driven with bias-corrected SST.
West African monsoon↗
Analytical noise bias correction for precise weak lensing shear inference
Noise bias is a significant source of systematic error in weak gravitational lensing measurements that must be corrected to satisfy the stringent standards of modern imaging surveys in the era of precision cosmology. This paper reviews the analytical noise bias correction method and provides analytical derivations demonstrating that we can recover shear to its second order using the ‘renoising’ noise bias correction approach introduced by METACALIBRATION. We implement this analytical noise bias correction within the AnaCal shear estimation framework and propose several enhancements to the noise bias correction algorithm. We evaluate the improved AnaCal using simulations designed to replicate Rubin Legacy Survey of Space and Time (LSST) imaging data. These simulations feature semi-realistic galaxies and stars, complete with representative distributions of magnitudes and Galactic spatial density. We conduct tests under various observational challenges, including cosmic rays, defective CCD columns, bright star saturation, bleed trails, and spatially variable point spread functions. Our results indicate a multiplicative bias in weak lensing shear recovery of less than a few tenths of a per cent, meeting LSST Dark Energy Science Collaboration requirements without requiring calibration from external image simulations. Additionally, our algorithm achieves rapid processing, handling one galaxy in less than a millisecond.
79 ASTRONOMY AND ASTROPHYSICS↗
Validation of the AMSU-B Bias Corrections Based on Satellite Measurements from SSM/T-2
The NOAA-15 Advanced Microwave Sounding Unit-B (AMSU-B) was designed in the same spirit as the Special Sensor Microwave Water Vapor Profiler (SSM/T-2) on board the DMSP F11-14 satellites, to perform remote sensing of spatial and temporal variations in mid and upper troposphere humidity. While the SSM/T-2 instruments have a 48 km spatial resolution at nadir and 28 beam positions per scan, AMSU-B provides an improvement with a 16 km spatial resolution at nadir and 90 beam positions per scan. The AMSU-B instrument, though, has been experiencing radio frequency interference (RFI) contamination from the NOAA-15 transmitters whose effect is dependent upon channel, geographic location, and current spacecraft antenna configuration. This has lead to large cross-track biases reaching as high as 100 Kelvin for channel 17 (150 GHz) and 50 Kelvin for channel 19 (183 +/-3 GHz). NOAA-NESDIS has recently provided a series of bias corrections for AMSU-B data starting from March, 1999. These corrections are available for each of the five channels, for every third field of view, and for three cycles within an eight second period. There is also a quality indicator in each data record to indicate whether or not the bias corrections should be applied. As a precursor to performing retrievals of mid and upper troposphere humidity, a validation study is performed by statistically analyzing the differences between the F14 SSM/T-2 and the bias corrected AMSU-B brightness temperatures for three months in the spring of 1999.
Kolodner, Marc A.↗
On the Limitations of Variational Bias Correction
Satellite radiances are the largest dataset assimilated into Numerical Weather Prediction (NWP) models, however the data are subject to errors and uncertainties that need to be accounted for before assimilating into the NWP models. Variational bias correction uses the time series of observation minus background to estimate the observations bias. This technique does not distinguish between the background error, forward operator error, and observations error so that all these errors are summed up together and counted as observation error. We identify some sources of observations errors (e.g., antenna emissivity, non-linearity in the calibration, and antenna pattern) and show the limitations of variational bias corrections on estimating these errors.
Moradi, Isaac↗