Search NASA⌕ Search

SEARCH · Search NASA

Results for “uncertainty in complex systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20
All sourcesNASA NTRSDOE OSTI
NASA NTRS2019

Hybrid Transportation System Integrated Trajectory Design and Optimization for Mars Landing Site Accessibility

NASA’s Mars Study Capability Team continues the agency’s efforts to study and refine the nation’s plan to field a sustainable human Mars campaign. One of the primary open issues left unanswered during previous analysis cycles was the uncertainty of the impact to the vehicle performance requirement to deliver crew and cargo to the same landing site across multiple mission opportunities. The Mars Study Capability Team has recently developed an integrated trajectory optimization and system closure model to solve the complex interplanetary trajectory optimization using both low-thrust and high-thrust maneuvers. This paper demonstrates the capability of this new integrated trajectory design and optimization method as it applies to the landing site accessibility problem for the Hybrid transportation architecture. The results showed that the current vehicle is capable of reaching up to +20 degree latitude and down to -20 degree latitude in every mission opportunity from 2033 to 2052. However, reaching latitudes beyond +/- 20 required more propellant than the spacecraft is currently designed to carry for many of the mission opportunities.

Chai, Patrick R.↗
NASA NTRS2015

Output-Based Adaptive Meshing Applied to Space Launch System Booster Separation Analysis

This paper presents details of Computational Fluid Dynamic (CFD) simulations of the Space Launch System during solid-rocket booster separation using the Cart3D inviscid code with comparisons to Overflow viscous CFD results and a wind tunnel test performed at NASA Langley Research Center's Unitary PlanWind Tunnel. The Space Launch System (SLS) launch vehicle includes two solid-rocket boosters that burn out before the primary core stage and thus must be discarded during the ascent trajectory. The main challenges for creating an aerodynamic database for this separation event are the large number of basis variables (including orientation of the core, relative position and orientation of the boosters, and rocket thrust levels) and the complex flow caused by the booster separation motors. The solid-rocket boosters are modified from their form when used with the Space Shuttle Launch Vehicle, which has a rich flight history. However, the differences between the SLS core and the Space Shuttle External Tank result in the boosters separating with much narrower clearances, and so reducing aerodynamic uncertainty is necessary to clear the integrated system for flight. This paper discusses an approach that has been developed to analyze about 6000 wind tunnel simulations and 5000 flight vehicle simulations using Cart3D in adaptive-meshing mode. In addition, a discussion is presented of Overflow viscous CFD runs used for uncertainty quantification. Finally, the article presents lessons learned and improvements that will be implemented in future separation databases.

Booster↗
NASA NTRS2018

Human-Automation Allocations for Current Robotic Space Operations: Space Station Remote Manipulator System

NASA’s Human Research Program’s Risk of Inadequate Design of Human and Automation/Robotic Integration (HARI) delineates the uncertainty surrounding crew work with automation and robotics in spaceflight. HARI is concerned with detrimental effects of ineffective user interfaces, system designs and/or functional task allocation on crew performance, potentially compromising mission success and safety. This risk arises because of limited experience with complex automation and robotics in spaceflight. One key knowledge gap within the HARI risk is related to function allocation.

Chang, Mai Lee↗
DOE OSTI2020

Multi-Model and Multi-Scale Global Sensitivity Analysis for Identifying Controlling Processes of Complex Systems

An environmental model consists of multiple process level sub-models, and each sub-model represents a process that is key to the operation of the simulated system. Global sensitivity analysis methods have been widely used to identify important processes for system model development and improvement. The existing methods of global sensitivity analysis only consider parametric uncertainty, and are not capable of handling model uncertainty caused by multiple process models that arise from competing hypotheses about one or more processes. To address this problem, this project develops a new method to probe model output sensitivity to competing process models by integrating model averaging methods with variance-based global sensitivity analysis to address uncertainty in process models and parameters. The new method yields three process sensitivity indices. The first one is called first-order process sensitivity index, and it is derived as a single summary measure of relative process importance. Evaluating the index is computationally expensive, because it relies in a Monte Carlo scheme that requires thousands and even millions of model executions. To reduce computational cost, this project develops a computationally efficient, quasi Monte Carlo method, and this method is presented in Chapter 2 of this report with and a numerical example for demonstration. The numerical example shows that the results of the quasi Monte Carlo method are substantially close to those of the full Monte Carlo method, but the computational cost of the quasi Monte Carlo method is only 0.7% of that of the full Monte Carlo method. The second index is called total-effect process sensitivity index, and it measures interactions between different processes. Therefore, this sensitivity index includes the first-order process sensitivity index, and can be used to identify influential processes. On the other hand, the total-effect process sensitivity index can also be used to screen non-influential processes. This is demonstrated by two numerical examples using the Sobol-G* functions and groundwater flow models that consider recharge process, geological process, and snowmelt process. The numerical examples shows that the total-effect process sensitivity index is more informative than the first-order process sensitivity. The derivation of the process sensitivity index and the numerical examples are discussed in Chapter 3. Chapter 4 presents two computationally efficient methods for screening non-influential processes to exclude them from further investigation. The two methods are the multi-model difference-based sensitivity (MMDS) analysis method, which can be implemented using the Latin Hypercube Sampling. The second one is the implementation of MMDS method using a binning method. The numerical example for the Sobol-G* function indicates the two methods are capable of identifying non-influential models, and the numerical examples for the groundwater flow and reactive transport show that the two methods are effective for groundwater problems. However, it should be noted that the two methods are numerical approximations, and they can only be used for screening non-influential processes, not for ranking importance of system processes. All the sensitivity analysis methods are implemented by developing python codes, and the codes are in a software called SAMMPY: a python package for process sensitivity analysis under multiple models. The SAMMPY design and structure are discussed in Chapter 5, and the package is released to the public for free download.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2021

Recent Advancements in Modeling and Simulation of Entry Systems at NASA

This paper describes recent development of modeling and simulation technologies for entry systems in support of NASA’s exploration missions. Mission-tailored research and development in modeling of entry systems occurs across the Agency (e.g., within the Orion and Mars 2020 Programs), however the aim of this paper is to discuss the broad, cross-mission research conducted by NASA’s Entry Systems Modeling (ESM) Project, which serves as the Agency’s only concerted effort toward advancing entry systems across a range of technical disciplines. Technology development in ESM is organized and prioritized from a system-level perspective, resulting in four broad technical areas of investment: (1) Predictive material modeling, (2) Shock layer kinetics and radiation, (3) Computational and experimental aerosciences, and (4) Guidance, navigation, and control. Investments in thermal protection material modeling are geared toward high-fidelity, predictive models capable of handling complex structures, with an eye toward optimizing design performance and quantifying thermal protection system reliability. New computational tools have been developed to characterize material properties and behavior at the microstructural level, and experimental techniques (molecular beam scattering, micro-computed tomography, among others) have been developed to measure material kinetics, morphology, and other parameters needed to inform and validate detailed simulations. Advancements have also been made in macrostructural simulation capability to enable 3-D system-scale calculations of material response with complex topological features, including differential recession of tile gaps. Research and development in the area of shock layer kinetics has focused on air and CO2-based atmospheres. Capacity and capability of the NASA Ames Electric Arc Shock Tube (EAST) have been expanded in recent years and analysis of resulting data has led to several improvements in kinetic models, while simultaneously reducing uncertainties associated with radiative heat transfer predictions. First-principles calculations of fundamental kinetic, thermodynamic, and transport data, along with state-specificmodels for non-equilibrium flow regimes, have also yielded new insights and have the potential to vastly improve model fidelity. Aerosciences is a very broad area of interest in entry systems, yet a number of important challenges are being addressed: Coupled fluid-structure simulations of parachute inflation and dynamics; Experimental and computational studies of vehicle dynamics; Multi-phase flow with dust particles to simulate entry environments at Mars during dust storms; Studies of roughness-induced heating augmentation relevant to tiled and woven thermal protection systems; and Advanced numerical methods to optimize computational analyses for desired accuracy versus cost. Guidance and control in the context of entry systems has focused on development of methods for multi-axis control (i.e. pitch and yaw, rather than bank angle alone) of spacecraft during entry and descent. With precision landing requirements driven by Mars human exploration goals, recent efforts have yielded 6-DOF models of multi-axis control with propulsive descent of both inflatable and rigid ellipsled-like architectures. Results for both configurations have demonstrated the ability to land within the 50-meter precision requirement demanded by Mars human exploration missions, while also reducing propellant requirements by enabling more efficient control through entry and descent. Ongoing research in GN&C is developing mechanical specifications for the systems and establishing engineering feasibility.

aerothermodynamics↗
NASA NTRS2019

Recent Advancements in Modeling and Simulation of Entry Systems at NASA

This paper describes recent development of modeling and simulation technologies for entry systems in support of NASA's exploration missions. Mission-tailored research and development in modeling of entry systems occurs across the Agency (e.g., within the Orion and Mars 2020 Programs), however the aim of this paper is to discuss the broad, cross-mission research conducted by NASA's Entry Systems Modeling (ESM) Project, which serves as the Agency's only concerted effort toward advancing entry systems across a range of technical disciplines. Technology development in ESM is organized and prioritized from a system-level perspective, resulting in four broad technical areas of investment: (1) Predictive material modeling, (2) Shock layer kinetics and radiation, (3) Computational and experimental aerosciences, and (4) Guidance, navigation, and control. Investments in thermal protection material modeling are geared toward high-fidelity, predictive models capable of handling complex structures, with an eye toward optimizing design performance and quantifying thermal protection system reliability. New computational tools have been developed to characterize material properties and behavior at the microstructural level, and experimental techniques (molecular beam scattering, micro-computed tomography, among others) have been developed to measure material kinetics, morphology, and other parameters needed to inform and validate detailed simulations. Advancements have also been made in macrostructural simulation capability to enable 3-D system-scale calculations of material response with complex topological features, including differential recession of tile gaps. Research and development in the area of shock layer kinetics has focused on air and CO2-based atmospheres. Capacity and capability of the NASA Ames Electric Arc Shock Tube (EAST) have been expanded in recent years and analysis of resulting data has led to several improvements in kinetic models, while simultaneously reducing uncertainties associated with radiative heat transfer predictions. First-principles calculations of fundamental kinetic, thermodynamic, and transport data, along with state-specific models for non-equilibrium flow regimes, have also yielded new insights and have the potential to vastly improve model fidelity. Aerosciences is a very broad area of interest in entry systems, yet a number of important challenges are being addressed: Coupled fluid-structure simulations of parachute inflation and dynamics; Experimental and computational studies of vehicle dynamics; Multi-phase flow with dust particles to simulate entry environments at Mars during dust storms; Studies of roughness-induced heating augmentation relevant to tiled and woven thermal protection systems; and Advanced numerical methods to optimize computational analyses for desired accuracy versus cost. Guidance and control in the context of entry systems has focused on development of methods for multi-axis control (i.e. pitch and yaw, rather than bank angle alone) of spacecraft during entry and descent. With precision landing requirements driven by Mars human exploration goals, recent efforts have yielded 6-DOF models of multi-axis control with propulsive descent of both inflatable and rigid ellipsled-like architectures.

Barnhardt, Michael D.↗
NASA NTRS2022

A Multimodel Investigation of Asian Summer Monsoon UTLS Transport Over the Western Pacific

The Asian summer monsoon (ASM) as a chemical transport system is investigated using a suite of models in preparation for an airborne field campaign over the Western Pacific. Results show that the dynamical process of anticyclone eddy shedding in the upper troposphere rapidly transports convectively uplifted Asian boundary layer air masses to the upper troposphere and lower stratosphere over the Western Pacific. The models show that the transported air masses contain significantly enhanced aerosol loading and a complex chemical mixture of trace gases that are relevant to ozone chemistry. The chemical forecast models consistently predict the occurrence of the shedding events, but the predicted concentrations of transported trace gases and aerosols often differ between models. The airborne measurements to be obtained in the field campaign are expected to help reduce the model uncertainties. Furthermore, the large-scale seasonal chemical structure of the monsoon system is obtained from modeled carbon monoxide, a tracer of the convective transport of pollutants, which provides a new perspective of the ASM circulation, complementing the dynamical characterization of the monsoon.

Laura L. Pan↗
DOE OSTI2024

Combining High-Throughput Experiments and Active Learning to Characterize Deep Eutectic Solvents

The high tunability of deep eutectic solvents (DESs) stems from the ease of changing their precursors and relative compositions. However, measuring the physicochemical properties across large composition and temperature ranges, necessary to properly design target-specific DESs, is tedious and error-prone and represents a bottleneck in the advancement and scalability of DES-based applications. As such, active learning (AL) methodologies based on Gaussian processes (GPs) were developed in this work to minimize the experimental effort necessary to characterize DESs. Owing to its importance for large-scale applications, the reduction of DES viscosity through the addition of a low-molecular-weight solvent was explored as a case study. A high-throughput experimental screening was initially performed on nine different ternary DESs. Then, GPs were successfully trained to predict DES viscosity from its composition and temperature, showcasing the ability of these stochastic, nonparametric models to accurately describe the physicochemical properties of complex mixtures. Finally, the ability of GPs to provide estimates of their own uncertainty was leveraged through an AL framework to minimize the number of data points necessary to obtain accurate viscosity modes. This led to a significant reduction in data requirements, with many systems requiring only five independent viscosity data points to be properly described.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

MSD CoP Webinar: "Generative agents: A new frontier for representing human actors and their behavior in MSD models"

Context: This webinar was hosted by the MultiSector Dynamics Community of Practice (MSD CoP; https://multisectordynamics.org). Talk #1: Behavioral Generative Agents for Energy Operations Presenter: Dr. Cong Chen (Thayer School of Engineering, Dartmouth College) Abstract: Accurately modeling consumer behavior in energy operations remains challenging due to inherent uncertainties, behavioral complexities, and limited empirical data. This talk introduces a novel approach leveraging generative agents--artificial agents powered by large language models--to realistically simulate customer decision-making in dynamic energy operations. Talk #2: Simulating multiple human perspectives in socio-ecological systems using large language models Presenter: Dr. Yongchao Zeng (Institute of Meteorology and Climate Research, Atmospheric Environmental Research (IMK-IFU) of the Karlsruhe Institute of Technology in Germany) Abstract: Understanding socio-ecological systems requires insights from diverse stakeholder perspectives. This talk describes a novel simulation system called HoPeS (Human-oriented Perspective Shifting). HoPeS enables model users to not only explore simulated socio-ecological systems (SESs) from a third-person observer's perspective but also take any of the simulated stakeholder roles, like playing an RPG game. By shifting multiple perspectives, model users can reflect and integrate the situated knowledge learned through the participatory simulation, approximating a more holistic and less biased understanding of SESs. Moderators: Jim Yoon (MSD CoP Human Systems Modeling Working Group Co-Chair); Stefano Galelli (MSD CoP Using AI to Enhance MSD Research Working Group Co-Chair); Patrick M. Reed (MSD CoP Facilitation Team) This webinar was held on: November 13th, 2025 from 12-1 PM EST.

Artificial Intelligence↗
NASA NTRS2003

The Aeronautical Data Link: Decision Framework for Architecture Analysis

A decision analytic approach that develops optimal data link architecture configuration and behavior to meet multiple conflicting objectives of concurrent and different airspace operations functions has previously been developed. The approach, premised on a formal taxonomic classification that correlates data link performance with operations requirements, information requirements, and implementing technologies, provides a coherent methodology for data link architectural analysis from top-down and bottom-up perspectives. This paper follows the previous research by providing more specific approaches for mapping and transitioning between the lower levels of the decision framework. The goal of the architectural analysis methodology is to assess the impact of specific architecture configurations and behaviors on the efficiency, capacity, and safety of operations. This necessarily involves understanding the various capabilities, system level performance issues and performance and interface concepts related to the conceptual purpose of the architecture and to the underlying data link technologies. Efficient and goal-directed data link architectural network configuration is conditioned on quantifying the risks and uncertainties associated with complex structural interface decisions. Deterministic and stochastic optimal design approaches will be discussed that maximize the effectiveness of architectural designs.

Morris, A. Terry↗
DOE OSTI2020

A Bayesian framework for adsorption energy prediction on bimetallic alloy catalysts

Abstract For high-throughput screening of materials for heterogeneous catalysis, scaling relations provides an efficient scheme to estimate the chemisorption energies of hydrogenated species. However, conditioning on a single descriptor ignores the model uncertainty and leads to suboptimal prediction of the chemisorption energy. In this article, we extend the single descriptor linear scaling relation to a multi-descriptor linear regression models to leverage the correlation between adsorption energy of any two pair of adsorbates. With a large dataset, we use Bayesian Information Criteria (BIC) as the model evidence to select the best linear regression model. Furthermore, Gaussian Process Regression (GPR) based on the meaningful convolution of physical properties of the metal-adsorbate complex can be used to predict the baseline residual of the selected model. This integrated Bayesian model selection and Gaussian process regression, dubbed as residual learning, can achieve performance comparable to standard DFT error (0.1 eV) for most adsorbate system. For sparse and small datasets, we propose an ad hoc Bayesian Model Averaging (BMA) approach to make a robust prediction. With this Bayesian framework, we significantly reduce the model uncertainty and improve the prediction accuracy. The possibilities of the framework for high-throughput catalytic materials exploration in a realistic setting is illustrated using large and small sets of both dense and sparse simulated dataset generated from a public database of bimetallic alloys available in Catalysis-Hub.org.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
NASA NTRSDate not supplied

Results of the NASA Prediction Uncertainty Reduction Tech Challenge

In January 2021 the Advanced Air Vehicles Program approved a new Tech Challenge (TC) to be run out of the Commercial Supersonic Technology (CST) Project. The objective of the TC was to bring uncertainties in the empirical noise modeling for civilian supersonic aircraft into the same range as that of conventional aircraft. The TC goal statement was to “Produce data and demonstrate tools that reduce the uncertainty in predicting Landing & Takeoff Noise levels of supersonic-relevant designs by 5 EPNdB and are ready for use in studies to inform FAA rule-making.” To make the required improvements in noise prediction methods for supersonic aircraft, NASA decided to validate and use high-fidelity numerical simulations to acquire the needed noise data on supersonic propulsion configurations. High-fidelity component designs were developed, such as a two-stage propulsor behind a supersonic inlet designed by General Electric Aviation and variable area exhaust systems using input from recent contracts with GE and Rolls Royce. A noise database was generated as a function of geometric and flow parameters, providing corrections to the empirical noise models and added new input variables to describe the complexities created by the supersonic-specific design features. Statistically, the new models were able to predict the acoustic impact of supersonic-specific features, the reduction in uncertainty being reduced from the baseline 7.8 EPNdB at the beginning of the TC to 2.0 EPNdB at the end.

noise prediction↗
DOE OSTI2025

Community-Informed Urban Flood Modeling for Impact Mitigation

The intensification of the hydrologic cycle due to climate change poses a threat to aging and under-designed water infrastructure systems which cannot adequately manage intense storm events. Developing a comprehensive plan for managing rain-driven flooding events is challenging due to uncertainties in the magnitude and frequency of future storm events and conflicting stakeholder objectives. In the City of Baltimore, Maryland, stormwater infrastructure is struggling to keep up with rainfall-driven (pluvial) flooding events, which regularly damage housing and disrupt transportation for residents. In this study, a hybrid of community engagement, numerical modeling, and artificial intelligence techniques are employed to explore prospective urban flooding adaptations. Community engagement drives the development of an urban flooding model (EPA Storm Water Management Model) for the Baltimore Harbor watershed. The model integrates complex surface and subsurface stormwater infrastructure data from the City, high-resolution spatial data, insights from local public works experts, and the lived experiences of City residents. This co-developed model simulates adaptations of interest to stakeholders in the city, including green and grey infrastructure and operational management strategies. Stormwater management scenarios focused on inlet cleaning and spatially concentrated green infrastructure are found to be the most effective in reducing flood depths in community priority locations. Together, these adaptations can reduce the duration of intersection inundation by more than twenty minutes, allowing for quicker emergency response and restoration of typical transportation systems. Future work will combine this community engaged flooding model with the Deep Uncertainties Pathways framework to explore tradeoffs between adaptations and develop dynamic adaptations which align with community objectives, enhance climate resilience in Baltimore, and can be adjusted in response to changing future conditions.

Ava, Spangler [Pennsylvania State University]↗
DOE OSTI2021

Time delay lens modelling challenge

ABSTRACT In recent years, breakthroughs in methods and data have enabled gravitational time delays to emerge as a very powerful tool to measure the Hubble constant H0. However, published state-of-the-art analyses require of order 1 yr of expert investigator time and up to a million hours of computing time per system. Furthermore, as precision improves, it is crucial to identify and mitigate systematic uncertainties. With this time delay lens modelling challenge, we aim to assess the level of precision and accuracy of the modelling techniques that are currently fast enough to handle of order 50 lenses, via the blind analysis of simulated data sets. The results in Rungs 1 and 2 show that methods that use only the point source positions tend to have lower precision ($10\!-\!20{{\ \rm per\ cent}}$) while remaining accurate. In Rung 2, the methods that exploit the full information of the imaging and kinematic data sets can recover H0 within the target accuracy (|A| < 2 per cent) and precision (<6 per cent per system), even in the presence of a poorly known point spread function and complex source morphology. A post-unblinding analysis of Rung 3 showed the numerical precision of the ray-traced cosmological simulations to be insufficient to test lens modelling methodology at the percent level, making the results difficult to interpret. A new challenge with improved simulations is needed to make further progress in the investigation of systematic uncertainties. For completeness, we present the Rung 3 results in an appendix and use them to discuss various approaches to mitigating against similar subtle data generation effects in future blind challenges.

Ding, X.↗
DOE OSTI2025

Geoanalytical Evaluation of Saline Storage (GEESS) Geodatabase v2.0

The Geoanalytical Economic Evaluation of Saline Storage (GEESS) geodatabase was developed to support the United States Department of Energy (DOE) and National Energy Technology Laboratory (NETL) in their geologic carbon storage efforts by characterizing saline geologic formations present in the FECM/NETL CO2 Saline Storage Cost Model (CO2_S_COM) [1]. Using publicly available literature and data, the GEESS geodatabase characterizes 57 geologic formations across the lower-48 U.S. states in what are called Fully Integrated Geodatabases (FIGs). The FIG is a vector polygon feature containing thousands or tens of thousands of individual polygons, which each contain discrete geologic parameter values. A list of the critical geologic parameters that are characterized in the GEESS geodatabase are described in the “Processing Steps and Workflow” part of the ReadMe file, as well as the Data Catalog accompanying the GEESS geodatabase. The FIG is the basis of the GEESS geodatabase and is the direct representation of the collected geologic data. In addition to the FIG, the GEESS system contains grid files. Due to the complexity of the FIGs, grids are used to sample the geologic data so they can be exercised within CO2_S_COM. The grid files contain the geologic data sampled from the FIG, as well as estimates of “Plume Uncertainty Diameter” and “First-year Break-even Price of CO2” derived from CO2_S_COM based on the GEESS grid data.

carbon↗
DOE OSTI2020

Characterization and Analysis of the Energy-Reporting Accuracy of Connected Devices

Emerging energy-efficient building systems increasingly exhibit greater functionality, often requiring multiple operating modes (e.g. white-tunability for lighting products and data traffic for devices with networked, integrated sensors). This increased functionality makes energy consumption estimates more complex. Given that these functions consume energy, the energy performance of such building systems is dependent on what operating modes they use and how much time they spend in each mode. Devices and systems that can report their own energy consumption mitigate this energy-performance uncertainty. This study explores the energy-reporting accuracy of market-available connected electrical outlets. The study considers two residential-market products (five units each, one outlet per unit) and three commercial-market products (two units each, 18 to 24 outlets per unit) with the ability to report power drawn and/or energy consumed by devices connected to their receptacles. The products were purchased through typical market channels. Pacific Northwest National Laboratory (PNNL) conducted testing in December 2018 at its Connected Lighting Test Bed (CLTB), using a custom-developed test setup and method adapted from industry standards. The setup collected energy-consumption data reported by the outlet devices under test (DUTs) at one-minute intervals and compared that data with measurements taken by a reference meter over a range of test conditions. The residential products reported power draw but not interval or cumulative energy consumption. The commercial products reported both power draw and cumulative energy consumption. Relative reporting error (RRE) was calculated for all measurements, and analysis of the results revealed variations across devices and test conditions. The total number of measurements (50 for each residential product, 60 for each commercial product) offers an appreciable comparison of performance at the make/model level. The average RRE of the residential products derived from reported power draw was -0.02% and -1.20%. The average RRE for two of the three the commercial products derived from reported power draw was worse than those of the residential products (-2.40%, -2.72%, -0.36%). The internal integration of power over time, used to calculate cumulative energy consumption, typically occurs at current and voltage sampling rates much higher than once per minute. This suggests that the average commercial-product RRE derived from reported energy consumption should be very consistent and better than performance based on reported power draw. However, the RRE derived from reported energy consumption varied significantly across the three makes of commercial-market products and was uniformly less accurate than performance based on reported power draw. Subsequent analysis identified a number of root causes for this decrease in performance, most of which were related to reporting resolution. The goals of this study are to generate awareness of building systems capable of reporting their own energy consumption, further interest in the value of energy data for a variety of uses, draw attention to how the accuracy of reported metrics can be characterized, and quantify the performance variation found in marketavailable products. The results of this study and subsequent related work may be relevant to stakeholders in industry-specification and standards-development organizations. The methods this study employs could inform test and measurement procedures and performance classifications for connected outlets, lighting products, and other building systems capable of reporting their own energy consumption. The study concludes with stakeholder recommendations, including the following: • Energy-reporting device and system manufacturers developing products that report energy consumption should characterize the accuracy of reported metrics using a reference meter calibrated by an independent laboratory that was accredited by an ILAC MRA signatory (and whose scope of accreditation explicitly covers energy measurement), and should include this information on product data sheets. • Standards and specification development organizations should develop application-specific performance classifications that end users can understand and relate to their energy-data use needs (e.g., 2% accuracy class for utility streetlight energy billing needs, or 10% accuracy class for ESCO performance verification needs). • Current or potential owners, operators, and specifiers of energy-reporting building systems should rigorously analyze the dependency of current and planned energy-data use cases on accuracy, noting in particular the dependence (or lack thereof) on relative vs. absolute accuracy, and on trueness vs. precision (i.e., repeatability), and should communicate use-case needs to industry standards and specification organizations.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2020

Characterization and Analysis of the Energy-Reporting Accuracy of Connected Devices

Emerging energy-efficient building systems increasingly exhibit greater functionality, often requiring multiple operating modes (e.g. white-tunability for lighting products and data traffic for devices with networked, integrated sensors). This increased functionality makes energy consumption estimates more complex. Given that these functions consume energy, the energy performance of such building systems is dependent on what operating modes they use and how much time they spend in each mode. Devices and systems that can report their own energy consumption mitigate this energy-performance uncertainty. This study explores the energy-reporting accuracy of market-available connected electrical outlets. The study considers two residential-market products (five units each, one outlet per unit) and three commercial-market products (two units each, 18 to 24 outlets per unit) with the ability to report power drawn and/or energy consumed by devices connected to their receptacles. The products were purchased through typical market channels. Pacific Northwest National Laboratory (PNNL) conducted testing in December 2018 at its Connected Lighting Test Bed (CLTB), using a custom-developed test setup and method adapted from industry standards. The setup collected energy-consumption data reported by the outlet devices under test (DUTs) at one-minute intervals and compared that data with measurements taken by a reference meter over a range of test conditions. The residential products reported power draw but not interval or cumulative energy consumption. The commercial products reported both power draw and cumulative energy consumption. Relative reporting error (RRE) was calculated for all measurements, and analysis of the results revealed variations across devices and test conditions. The total number of measurements (50 for each residential product, 60 for each commercial product) offers an appreciable comparison of performance at the make/model level. The average RRE of the residential products derived from reported power draw was -0.02% and -1.20%. The average RRE for two of the three the commercial products derived from reported power draw was worse than those of the residential products (-2.40%, -2.72%, -0.36%). The internal integration of power over time, used to calculate cumulative energy consumption, typically occurs at current and voltage sampling rates much higher than once per minute. This suggests that the average commercial-product RRE derived from reported energy consumption should be very consistent and better than performance based on reported power draw. However, the RRE derived from reported energy consumption varied significantly across the three makes of commercial-market products and was uniformly less accurate than performance based on reported power draw. Subsequent analysis identified a number of root causes for this decrease in performance, most of which were related to reporting resolution. The goals of this study are to generate awareness of building systems capable of reporting their own energy consumption, further interest in the value of energy data for a variety of uses, draw attention to how the accuracy of reported metrics can be characterized, and quantify the performance variation found in marketavailable products. The results of this study and subsequent related work may be relevant to stakeholders in industry-specification and standards-development organizations. The methods this study employs could inform test and measurement procedures and performance classifications for connected outlets, lighting products, and other building systems capable of reporting their own energy consumption. The study concludes with stakeholder recommendations, including the following: • Energy-reporting device and system manufacturers developing products that report energy consumption should characterize the accuracy of reported metrics using a reference meter calibrated by an independent laboratory that was accredited by an ILAC MRA signatory (and whose scope of accreditation explicitly covers energy measurement), and should include this information on product data sheets. • Standards and specification development organizations should develop application-specific performance classifications that end users can understand and relate to their energy-data use needs (e.g., 2% accuracy class for utility streetlight energy billing needs, or 10% accuracy class for ESCO performance verification needs). • Current or potential owners, operators, and specifiers of energy-reporting building systems should rigorously analyze the dependency of current and planned energy-data use cases on accuracy, noting in particular the dependence (or lack thereof) on relative vs. absolute accuracy, and on trueness vs. precision (i.e., repeatability), and should communicate use-case needs to industry standards and specification organizations.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗