Climate Model Benchmarking: Building Trust and Advancing Science
Explore the source record for details and available documents.
Hassler, Birgit [German Aerospace Center (DLR) Obe↗
SEARCH · Search NASA
Results for “Building benchmarking models”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Beyond Energy Efficiency: A clustering approach to embed demand flexibility into building energy benchmarking
The intermittency of carbon-free renewables and the demand changes associated with the widespread push for electrifying the transportation and building sectors provides an opportunity for buildings to go beyond energy efficiency and push towards providing demand flexibility to the electricity grid. The duality of energy efficiency and demand flexibility is necessary for success in a sustainable and reliable energy transition. Current building energy benchmarking models are limited in their ability to integrate concepts of demand flexibility and/or utilize granular smart meter data. Thus, current benchmarking methods are focused annual energy usage and fail to incorporate how the time of use of energy consumption impacts emissions in a quickly changing energy grid. Without a more comprehensive view of energy usage and associated real-time emissions, current benchmarking methods are unlikely to realize the full decarbonization potential of buildings. New emerging data streams provide an opportunity to develop a new generation of benchmarking energy models that embed dimensions of energy efficiency, grid interactivity, and demand flexibility into their analysis. In this paper, we propose a four-step method for embedding grid interactivity and demand flexibility into building benchmarking models that utilizes emerging building and time-series electricity data streams. We first engineer features to produce a mix-type dataset that encompasses many attributes of grid-interactive and efficient buildings, and then we apply K-medoids using Gower's Distance to produce peer-group clusters. We apply the method to a case study of 306 primary and secondary schools in southern California, USA. The results show that the method effectively clusters buildings by attributes of demand flexibility and energy efficiency. The clustering results reveal patterns in inefficient building operations and demand inflexibility at the building peer group level. In conclusion, the interpretation of clusters can serve as an integrated energy efficiency and demand flexibility benchmarking model and inform performance-specific policy targeting for buildings that go beyond traditional efficiency measures.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Surrogate model evaluation and building energy benchmarking for commercial buildings
Building energy consumption benchmarking involves challenges associated with various energy patterns for different building types; heating, ventilating, and air-conditioning (HVAC) system types; and climates. Given significant variation in energy use patterns, accurate prediction of long-term energy use using surrogate models remains challenging. Multiple linear regression (MLR) is commonly used for building energy benchmarking because of its simple structure; however, it lacks accuracy compared to other black-box models. Although many studies have compared surrogate models and offer guidance on model selection based on metrics, they do not provide detailed analysis on improving the surrogate model accuracy. In this paper, we implement a surrogate model using polynomial ridge regression (i.e., MLR with interaction terms combined with ridge regularization) for small office and retail strip mall buildings across six HVAC system types and all climate zones, for electricity and natural gas in baseline and proposed scenarios. A simulation workflow is developed using OpenStudio TM /EnergyPlus TM to generate simulation data using measures over a wide range of efficiency inputs. Enhancements based on statistical insights are used for improving the model accuracy using filters, input transformations, and change points. Surrogate models achieved average coefficient of variation of the root mean squared error (CVRMSE) values of 2.17, 1.06, 2.05, and 3.26 for proposed electricity, proposed natural gas, baseline electricity, and baseline natural gas, respectively, with enhancements reducing CVRMSE by an average of 14.9% across all combinations. We provide model interpretation via Shapley additive explanations to determine which input variables most influence energy consumption and provide supportive arguments for enhancements.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
Community Geothermal: Energy, Cost, and Carbon Modeling for District Design - Ann Arbor, MI
This data includes results on an analysis of existing and projected energy, cost, and carbon for the City of Ann Arbor - District Geothermal Design and Deployment to Equitably Decarbonize Low Income Neighborhoods in Ann Arbor project. The scope of the project includes designing and implementing a geothermal district heating and cooling system that reduces thermal heating and cooling load by 75% and greenhouse gas emissions by 40% in the project area (262 households, 6 commercial buildings). The existing neighborhood was modeled using Design Builder, an EnergyPlus software, to understand the current energy load. The energy model was then flipped to reflect the designed district geothermal heating and cooling system to project the effect on energy, carbon, and cost. This dataset includes the analysis files utilized and created for this study. There are 3 categories of data: 1) existing/benchmarking, 2) energy modeling, and 3) post processed calculations. This follows the methodology and process of the project team, which is fully explained in file 00_Technical Economic Environmental Assessment. All uses of data are referenced throughout this assessment to their respective files included below.
15 GEOTHERMAL ENERGY↗
AI Model Benchmarking for Nonproliferation Applications: Steel Thread Benchmarking Task Force Technical Report (Rev. 2)
Steel Thread is a NA-22 venture that seeks to build trustworthy, reliable AI models that can be used in a wide variety of nonproliferation tasks. A key aspect of building these models is developing appropriate benchmarks and evaluation methods, which will enable the venture to identify and adapt models to provide the most value in the nonproliferation domain. Benchmarks must be relevant to key tasks in this domain, such as question answering, information retrieval, document summarization and classification, consensus analysis, and image and data analysis. This report 1) provides an overview of benchmark design, evaluation, and challenges; 2) reviews a variety of open benchmarks, with a focus on language models and tasks; and 3) identifies benchmarks that are most relevant to Steel Thread. This report is intended to serve as a basis for further efforts to classify and evaluate benchmarks and their correlation with success on nonproliferation-specific tasks. The Steel Thread venture has defined benchmarks to be a particular combination of a dataset (or datasets) and a metric (or metrics) conceptualized as representing one or more specific tasks or sets of abilities for a specific modality. It is adopted by a research community as a shared framework for comparing methods.1 It includes 1) Data: Labeled (a designated subset not used for training, which could be all the data), 2) Metric: A way to quantify performance, 3) Task/Ability: What the benchmark is testing, 4) Protocol: A structured and repeatable evaluation process, 5) Baseline/Reference Model: For comparison; could be statistical, rule-based, SME-derived, or another model, and 6) Maintenance Plan: to update with new information over time; important for long-term utility. For further clarity, the definition includes what a benchmark, in this context, is not. It is not a corpus of training data, specific to a model (it is intended to apply to a range of models), a universal evaluation of performance, a guarantee that the ‘top’ model on the leaderboard will be the best fit for every specific use case, an all-encompassing proof of a model’s universal quality, nor is it a one-size-fits-all measure of success. It does not cover every real-world constraint (like operational, ethical, or cost considerations), a systems integration test, or a unit test. This definition was inspired by and resulted from discussions within the Steel Thread Benchmarking Task Force. This group was formed to define what we would mean as a benchmark within Steel Thread but persisted as the need to develop a thorough understanding of the large and expanding existing benchmarking space. This technical report is a result of the group’s divide and conquer approach to exploring this space. The release of benchmarks might not be progressing as quickly as model development, but it is moving very fast, as many benchmarks quickly become saturated, when state-of-the-art models score so close to the benchmark’s ceiling that their results are virtually indistinguishable. At that point, the test no longer differentiates between new systems, so researchers usually stop reporting scores as the benchmark no longer informs about improvements from the next generation of models. In the OpenAI announcement of GPT-5, they reported results on six flagship public benchmarks (AIME 2025, SWE-bench Verified, Aider Polyglot, MMMU, HealthBench Hard, GPQA) but the full system-card covers roughly thirty-five separate evaluations, comprising hundreds of test task items in total. There have been some efforts to summarize benchmarks in specific fields, like for text-to-image generation, but these surveys have had a narrow methodology scope. Therefore, a comprehensive survey of all benchmarks or even all benchmarks that could be relevant to Steel Thread is outside of the scope of this report. We chose some specific benchmarks to investigate in detail.
97 MATHEMATICS AND COMPUTING↗
Multi-criteria thermal resilience certification scheme for indoor built environments during heat waves
With climate change, the indoor built environment is expected to significantly influence the occupant's safety and well-being. A novel multi-criteria thermal resilience certification scheme for indoor built environments during extreme heat events is proposed in this paper. The certification scheme considers overheating, thermal comfort, heat stress, and hygrothermal discomfort in built environments. These criteria are quantified using key performance indicators like indoor overheating degree, hours of exceedance, wet-bulb globe temperature, and heat index, respectively. This scheme is developed based on existing best practices like standards, rating systems, and literature. The scheme is implemented on a benchmark building energy performance model for detached post-World War II dwellings in Belgium as a case study using weather data measured from the City of Brussels. The indoor overheating in the reference dwelling is assessed with a static threshold of 27 °C for the bedrooms and adaptive thresholds for other areas. The analysis found that the building performance is within the defined threshold levels throughout the heat wave duration for all criteria. Therefore, the reference dwelling got a maximum attainable score of four points and is rated five-star for thermal resilience during heat waves. The proposed certification scheme is intended as a standardized framework and highlights the need for further revisions in building performance policies and guidelines.
Climate change↗
Security-Constrained Unit Commitment for Electricity Market: Modeling, Solution Methods, and Future Challenges
This paper summarizes the technical activities of the IEEE Task Force on Solving Large Scale Optimization Problems in Electricity Market and Power System Applications. This Task Force was established by the IEEE Technology and Innovation Subcommittee to first review the state-of-the-art of the security-constrained unit commitment (SCUC) business model, its mathematical formulation, and solution techniques in solving electricity market clearing problems. The Task Force then investigated the emerging challenges of future market clearing problems and presented efforts in building benchmark mathematical and business models.
24 POWER TRANSMISSION AND DISTRIBUTION↗
A synthetic building operation dataset
Abstract This paper presents a synthetic building operation dataset which includes HVAC, lighting, miscellaneous electric loads (MELs) system operating conditions, occupant counts, environmental parameters, end-use and whole-building energy consumptions at 10-minute intervals. The data is created with 1395 annual simulations using the U.S. DOE detailed medium-sized reference office building, and 30 years’ historical weather data in three typical climates including Miami, San Francisco, and Chicago. Three energy efficiency levels of the building and systems are considered. Assumptions regarding occupant movements, occupants’ diverse temperature preferences, lighting, and MELs are adopted to reflect realistic building operations. A semantic building metadata schema - BRICK, is used to store the building metadata. The dataset is saved in a 1.2 TB of compressed HDF5 file. This dataset can be used in various applications, including building energy and load shape benchmarking, energy model calibration, evaluation of occupant and weather variability and their influences on building performance, algorithm development and testing for thermal and energy load prediction, model predictive control, policy development for reinforcement learning based building controls.
24 POWER TRANSMISSION AND DISTRIBUTION↗
AlphaBuilding - Synthetic Buildings Operation Dataset
This is a synthetic building operation dataset which includes HVAC, lighting, miscellaneous electric loads (MELs) system operating conditions, occupant counts, environmental parameters, end-use and whole-building energy consumptions at 10-minute intervals. The data is created with 1395 annual simulations using the U.S. DOE detailed medium-sized reference office building, and 30 years' historical weather data in three typical climates including Miami, San Francisco, and Chicago. Three energy efficiency levels of the building and systems are considered. Assumptions regarding occupant movements, occupants' diverse temperature preferences, lighting, and MELs are adopted to reflect realistic building operations. A semantic building metadata schema - BRICK, is used to store the building metadata. The dataset is saved in a 1.2 TB of compressed HDF5 file. This dataset can be used in various applications, including building energy and load shape benchmarking, energy model calibration, evaluation of occupant and weather variability and their influences on building performance, algorithm development and testing for thermal and energy load prediction, model predictive control, policy development for reinforcement learning based building controls.
AlphaBuilding↗
Building Efficiency Targeting Tool for Energy Retrofits (BETTER) Web Application (BETTER Web App) v1.0
The Building Efficiency Targeting Tool for Energy Retrofits (BETTER) V.1.0 web application identifies cost-saving energy and emissions reductions in buildings and portfolios, without site visits or complex modeling. With minimal data entry, BETTER benchmarks a building's or portfolio's energy use against peers; quantifies energy, cost, and greenhouse gas (GHG) reduction potential; and recommends energy efficiency measures for individual buildings or portfolios, targeting specific energy savings levels. The source code of BETTER's modular, cross-platform analytical engine has previously been disclosed (2019-001) and is available on GitHub and can be adopted, redeveloped, and redistributed freely under an open-source license, allowing users to incorporate BETTER's analytical capabilities into their own software platforms and tools. The BETTER V.1.0 web application, developed with the Django web-framework and the Model-View-Controller (MVC) architecture being disclosed here, provides a graphical user interface (GUI) for any user to view the software tutorials, download and upload a data entry template, configure and run the BETTER analyses, and view the final analytical reports.
Szum, Carolyn↗
Physics vs structure: A systematic benchmark of learning strategies for multi-zone building thermal dynamics
Recent advances in physics-informed and data-driven machine learning promise improved thermal models for advanced building control, yet there is limited quantitative evidence on when added physics structure and architectural complexity are beneficial. Here, this work presents a systematic benchmark of five representative system identification methods for modeling multi-zone building thermal dynamics: linear state-space models, multi-layer perceptrons, neural state-space models, neural ordinary differential equations, and physically-consistent neural networks. The methods are evaluated across multiple data regimes and zone coupling strategies. Using a high-fidelity multi-zone commercial building emulator, we examine short-term and long-term prediction accuracy, computational efficiency, and ease of development. Our results reveal critical trade-offs between prediction performance, model complexity, and physical consistency. We demonstrate that decoupled, nonlinear black-box models consistently outperform coupled physics-constrained architectures in both predictive accuracy and out-of-distribution robustness in majority of the test cases for the building type considered in the study. Our findings quantify the cost of complexity in building thermal modeling and provide concrete, actionable, scenario-based guidelines for selecting model classes for control-oriented applications.
Building thermal modeling↗
Performance analysis and comparison of data-driven models for predicting indoor temperature in multi-zone commercial buildings
Building thermal models, which characterize the properties of a building’s envelope and thermal mass, are essential for accurate indoor temperature and cooling/heating demand prediction. Because of their flexibility and ease of use, data-driven models are increasingly used. Here, this study compared and analyzed the performance of gray-box (resistance-capacitance) and black-box (recurrent neural network) models for predicting indoor air temperature in a real multi-zone commercial building. The developed resistance-capacitance model served as a benchmark model for which full sets of temporal data and building information were used as inputs. The recurrent neural network models were trained and tested assuming various available types and amounts of temporal data and known building physical information to investigate the effects of data and information availability. Feature importance analysis was conducted to select the key variables for different prediction targets under different scenarios. This research provides guidance in selecting an appropriate building thermal response modeling method based on the measured data availability, building physical information, and application.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
A general spatial-temporal framework for short-term building temperature forecasting at arbitrary locations with crowdsourcing weather data
Weather forecasting has been a critical component to predict and control building energy consumption for better building energy management. Without accessibility to other data sources, the onsite observed temperatures or the airport temperatures are used in forecast models. In this paper, we present a novel approach by utilizing the crowdsourcing weather data from neighboring personal weather stations (PWS) to improve the weather forecast accuracy around buildings using a general spatial-temporal modeling framework. The final forecast is based on the ensemble of local forecasts for the target location using neighboring PWSs. Our approach is distinguished from existing literature in various aspects. First, we leverage the crowdsourcing weather data from PWS in addition to public data sources. In this way, the data is at much finer time resolution (e.g., at 5-minute frequency) and spatial resolution (e.g., arbitrary location vs grid). Second, our proposed model incorporates spatial-temporal correlation information of weather variables between the target building and a set of neighboring PWSs so that underlying correlations can be effectively captured to improve forecasting performance. Here, we demonstrate the performance of the proposed framework by comparing to the benchmark models on temperature forecasting for a building located at an arbitrary location at San Antonio, Texas, USA. In general, the proposed model framework equipped with machine learning technique such as Random Forest can improve forecasting by 50% compares with persistent model and has 90% chance to outperform airport forecast in short-term forecasting. In a real-time setting, the proposed model framework can provide more accurate temperature forecasting results compared with using airport temperature forecast for most forecast horizon. Moreover, we analyze the sensitivity of model parameters to gain insights on how crowdsourcing data from the neighboring personal weather stations impacts forecasting performance. Finally, we implement our model in other cities such as Syracuse and Chicago to test the model's performance in different landforms and climate types.
54 ENVIRONMENTAL SCIENCES↗
CityBES v2021
City Buildings, Energy, and Sustainability (CityBES) is a web-based data and computing platform, focusing on energy modeling and analysis of a city's building stock to support district or city-scale building energy efficiency programs. CityBES uses an international open data standard, CityGML, to represent and exchange 3D city models. CityBES employs EnergyPlus to simulate building energy use and savings from energy efficient retrofits. Other CityBES features include energy benchmarking, district heating and cooling system modeling, rooftop PV analysis, building performance visualization, heat resilience modeling, as well as urban scale mapping of microclimate and heat vulnerability at census tract level. Different from other tools, CityBES uses integrated open and standard 3D city building data and models each individual building using EnergyPlus. CityBES can be used by urban planners, city energy managers, building owners, utilities, energy consultants and researchers.
Hong, Tianzhen↗
U.S.-China Clean Energy Research Center Building Energy Efficiency (CERC-BEE) Open-Source Retrofit Targeting Tool (CRADA FP00007338 Final Report)
To increase the cost-saving energy and carbon dioxide (CO 2 ) emissions reductions in buildings and portfolios at the scale and speed necessary to limit climate change, researchers at LBNL and Johnson Controls (JCI) developed the Building Efficiency Targeting Tool for Energy Retrofits (BETTER). BETTER is a software tool that consists of three components: (1) the BETTER analytical engine source code (which was developed with intellectual property provided by JCI under CRADA FP00007338); (2) the BETTER web application, developed by LBNL and McQuillen Interactive Pty. Ltd; and (3) the BETTER application programming interface (API), also developed by LBNL and McQuillen Interactive Pty. Ltd. BETTER enables building and portfolio owners, managers, and service providers worldwide to quickly, easily identify cost-saving energy efficiency retrofits in existing buildings and portfolios without expensive site visits or complex modeling. With minimal data input, the tool benchmarks a building’s electric and fossil energy usage against peers; quantifies energy, cost and greenhouse gas (GHG) emission reduction potentials at the building and portfolio levels; and recommends energy efficiency measures to decarbonize and electrify buildings and portfolios, targeting specific energy savings levels. No other tool so comprehensively analyzes buildings and portfolios with such ease. If fully implemented, it is estimated that BETTER could help reduce emissions equivalent to planting 1.3 billion trees globally by 2030. Moreover, an additional 50-75% of embodied GHG emissions could be avoided in each case where BETTER results in a building being retrofitted instead of demolished and replaced, providing substantial additional decarbonization benefits for the buildings sector. BETTER has garnered multiple awards and avid interest from investors. In 2020, it earned a R&D 100 Award for innovation and a LBNL Director’s Award for Technology Transfer. In 2021, BETTER was named an EarthX E-Capital Summit Climate Tech Prize semi-finalist
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
Advancing AI-Driven Analysis in X-ray Absorption Spectroscopy: Spectral Domain Mapping and Universal Models
In recent years, rapid progress has been made in developing artificial intelligence (AI) and machine learning (ML) methods for X-ray absorption spectroscopy (XAS) analysis. Compared to traditional XAS analysis methods, AI/ML approaches offer dramatic improvements in efficiency and help eliminate human bias. To advance this field, we advocate an AI-driven XAS analysis pipeline that features several interconnected key building blocks: benchmarks, workflows, databases, and AI/ML models. Specifically, we present two case studies for XAS ML. In the first study, we demonstrate the importance of reconciling the discrepancies between simulation and experiment using spectral domain mapping (SDM). Our ML model, which is trained solely on simulated spectra, predicts an incorrect oxidation state trend for Ti atoms in a combinatorial zinc titanate film. After transforming the experimental spectra into a simulation-like representation using SDM, the same model successfully recovers the correct oxidation state trend. In the second study, we explore the development of universal XAS ML models that are trained on the entire periodic table, which enables them to leverage common trends across elements. Looking ahead, we envision that an AI-driven pipeline can unlock the potential of real-time XAS analysis to accelerate scientific discovery.
36 MATERIALS SCIENCE↗
Utilizing ensemble learning for performance and power modeling and improvement of parallel cancer deep learning CANDLE benchmarks
Abstract Machine learning (ML) continues to grow in importance across nearly all domains in modeling to learn from data. Often a tradeoff exists between a model's ability to minimize bias and variance. In this article, we utilize ensemble learning to combine linear, nonlinear, and tree‐/rule‐based ML methods to cope with the bias‐variance tradeoff and result in more accurate models. We use the datasets collected for two parallel cancer deep learning CANDLE benchmarks, NT3 and P1B2, to build performance and power models based on hardware performance counters using single‐object and multiple‐objects ensemble learning to identify the most important counters for improvement on the Cray XC40 Theta at Argonne National Laboratory. Based on the insights from these models, we improve the performance and energy of P1B2 and NT3 by optimizing the deep learning environments TensorFlow, Keras, Horovod, and Python under the huge page size of 8 MB. Experimental results show that ensemble learning not only produces more accurate models but also provides more robust performance counter ranking. We achieve up to 61.15% performance improvement and up to 62.58% energy saving for P1B2 and up to 55.81% performance improvement and up to 52.60% energy saving for NT3 on up to 24,576 cores.
Wu, Xingfu↗
Direct Observations of Solute Dispersion in Rocks With Distinct Degree of Sub‐Micron Porosity
Abstract The transport of chemical species in rocks is affected by their structural heterogeneity to yield a wide spectrum of local solute concentrations. To quantify such imperfect mixing, advanced methodologies are needed that augment the traditional breakthrough curve analysis by probing solute concentration within the fluids locally. Here, we demonstrate the application of asynchronous, multimodality imaging by X‐ray computed tomography (XCT) and positron emission tomography (PET) to the study of passive tracer experiments in laboratory rock cores. The four‐dimensional concentration maps measured by PET reveal specific signatures of the transport process, which we have quantified using fundamental measures of mixing and spreading. We observe that the extent of solute spreading correlate strongly with the strength of subcore‐scale porosity heterogeneity measured by XCT, while dilution is enhanced in rocks containing substantial sub‐micron porosity. We observe that the analysis of different metrics is necessary, as they can differ in their sensitivity to the strength and forms of heterogeneity. The multimodality imaging approach is uniquely suited to probe the fundamental difference between spreading and mixing in heterogeneous media. We propose that when multi‐dimensional data is available, mixing and spreading can be independently quantified using the same metric. We also demonstrate that one‐dimensional transport models have limited predictive ability toward the internal evolution of the solute concentration, when the model is solely calibrated against the effluent breakthrough curves. The data set generated in this study can be used to build realistic digital rock models and to benchmark transport simulations that account deterministically for rock property heterogeneity.
Kurotori, Takeshi [Department of Chemical Engineer↗