Evaluating the quality of uncertainty quantification enabled deep learning models.
Abstract not provided.
Ries, Daniel↗
SEARCH · Search NASA
Results for “Quality models”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Monitoring water quality in the lower Kansas River using remote sensing
Abstract We demonstrate how to combine remote sensing data from satellite imagery (Sentinel‐2) with in situ water quality gauging (USGS Super Gages and the Gybe hyperspectral radiometer) to create spatially dense maps of water quality parameters (chlorophyll‐a concentration, turbidity, and nitrate plus nitrite concentration) along the lower Kansas River. The water quality maps are created using locally tuned models of the target water quality parameters, and this study describes the steps used to design, calibrate, and validate the empirical correlations. Water quality parameters such as chlorophyll‐a concentration are correlated with well‐studied absorption and scattering features in the visible spectrum (roughly 400–700 nm). Nutrients (such as nitrate plus nitrite concentration) lack strong absorption features in the visible spectrum, and in those cases we describe a novel surrogate data modeling approach that identifies overlapping water parcels between the in situ gauging and the remote sensing imagery. Measurements from the overlapping water parcels yield excellent correlations () for the target water quality parameters for limited windows of time (or limited sections of river reaches). Examples are provided illustrating how the water quality maps can be used to track river inputs from ungauged sources (such as creeks), or reveal the mixing patterns at the confluences.
Tufillaro, Nicholas↗
Mapping wall-to-wall fractional cover of Arctic tundra plant functional types in Alaska using 20-m spatial resolution satellite imagery and harmonized plot observations
Estimates of fractional cover (fCover) across given land surfaces are used to assess, and often model, vegetation composition and diversity, which are crucial for understanding the health and functioning of terrestrial ecosystems. Remote sensing provides a useful means for scaling local, plot-measured fCover estimates to regional scales. Leveraging a recently synthesized and harmonized plot database, this study generated wall-to-wall maps of fCover for six Alaskan-Arctic plant functional types (PFT), including non-vascular plants, forbs, graminoids, and deciduous and evergreen shrubs, using 20-m satellite data (Sentinel-1, Sentinel-2, ArcticDEM) using a machine learning regression approach, specifically the random forest (RF) algorithm, which is well-suited for handling nonlinear relationships and high-dimensional satellite datasets. This study additionally addressed the spatio-temporal inconsistencies e.g., sampling scale, plot size, and collection year in plot measured fCover by adopting a multivariate outlier detection approach—Cook’s distance—to identify high-quality plots for model training and validation. Our approach achieves high accuracy (R 2 = 0.59–0.93, root mean squared errors = 0.02–0.10 for all PFTs) between plot-observed and satellite-derived fCover when using high-quality plot samples. The mapped fCover characterizes the spatial patterns of different PFTs across the tundra biome at a 20-m resolution, providing key information needed for improved representation of Arctic tundra vegetation in terrestrial biosphere models to better understand climate-vegetation feedback across the Arctic tundra.
Arctic tundra↗
Benchmark Model for Wastewater Treatment Using an Activated Sludge Process
This is benchmark model for wastewater treatment using an activated sludge process. The activated sludge process is a means of treating both municipal and industrial wastewater. The activated sludge process is a multi-chamber reactor unit that uses highly concentrated microorganisms to degrade organics and remove nutrients from wastewater, producing quality effluent. This model provides pollutant concentrations, mass balance, electricity requirements, and treatment costs. This model will be continuously updated based on the latest data.
29 ENERGY PLANNING, POLICY, AND ECONOMY↗
Rapid Annotation of Photosynthetic Systems (RAPS): automated algorithm to generate genome-scale metabolic networks from algal genomes
Algae have mounting potential to serve as platform strains for engineering efforts, but tools for these species lag behind traditional model systems like E. coli and yeast. Metabolic models are one such tool that would enhance our ability to rationally engineer algae but current automated algorithms to build models do not perform well for algal genomes. We introduce a novel software pipeline, Rapid Annotation of Photosynthetic Systems (RAPS), which leverages manual curation efforts of published models to create high quality first draft metabolic networks for new species. We compared models produced by our pipeline to published models and found that these models are able to capture more genes than the published models and are more predictive of experimentally determined growth rate. We used RAPS to automate the creation of 8 first draft metabolic models of new species to enable the further study of metabolism in these species.
09 BIOMASS FUELS↗
A Short-Term Solar Forecasting Platform Using a Physics-Based Smart Persistence Model and Data Imputation Method
Electrical energy plays vital role in our socio-economic activity and therefore ensuring the reliability of the electric grid, from the generation, transmission and distribution level is critical. In order to maintain the power system parameter viz., frequency, voltage, etc., optimally, balancing of generation and consumption is very much essential. However, solar energy is infirm power by nature this is due to cloud cover / other local phenomena. Hence, Photovoltaic (PV) power generation brings a significant challenge to the grid operator due to the variability of the solar energy. The complexity of this challenge in terms of planning and dispatch ability of PV resources, aggravates with the high penetration of solar energy into the electric grid. In this setting, reliable solar radiation forecasting models based on accurate and quality input data become essential. In order to develop a suitable model for predicting solar radiation, quality historical / real time measurement is also needed. Under this study NIWE and NREL jointly developed / tested short-term solar forecasting frameworks using a smart persistence and physics-based smart persistence models for intra-hour forecasting of solar radiation (PSPI) and benchmarked 9 different data imputation techniques in 15 Solar Radiation Resource Assessment (SRRA) stations, located at different parts of India. During any measurement campaign, due to various technical reasons, we may miss few observations. However, the missing observation often reduce the performance of any forecasting model. Therefore, suitable data imputation method would assist us to obtain continuous observation of solar radiation. A station-by-station and method-by-method analysis was carried out to understand the performance of each model. Based on our analysis, among all the data imputation methods, the Kalman data imputation method is better for Indian Weather condition. In addition, Kalman StructTS, Linear, Stine and Arima methods yield slightly inferior accuracy compared to Kalman, but outperform the other methods. The extended solar radiation data are used by solar forecasting models to provide the prediction of solar radiation at 15 SRRA stations. As far as short term forecasting model is concerned, the PSPI model outperforms the Smart Persistence model. However, the forecast error is increases with the forecasting horizon.
29 ENERGY PLANNING, POLICY, AND ECONOMY↗
Modeling O and N Alloying in Nb for SRF Applications
N and O-alloyed superconducting radio frequency cavities exhibit extraordinary quality factors. Developing diffusion models that describe interstitial N and O in Nb is important for optimizing alloyed cavity quality factors and accelerating gradients. N and O-alloyed Nb samples are examined with SEM AND SIMS and their diffusion profiles modeled.
Lechner, Eric↗
Improving and Assessing the Quality of Uncertainty Quantification in Deep Learning
Deep learning (DL) models have enjoyed increased attention in recent years because of their powerful predictive capabilities. While many successes have been achieved, standard deep learning methods suffer from a lack of uncertainty quantification (UQ). While the development of methods for producing UQ from DL models is an active area of current research, little attention has been given to the quality of the UQ produced by such methods. In order to deploy DL models to high-consequence applications, high-quality UQ is necessary. This report details the research and development conducted as part of a Laboratory Directed Research and Development (LDRD) project at Sandia National Laboratories. The focus of this project is to develop a framework of methods and metrics for the principled assessment of UQ quality in DL models. This report presents an overview of UQ quality assessment in traditional statistical modeling and describes why this approach is difficult to apply in DL contexts. An assessment on relatively simple simulated data is presented to demonstrate that UQ quality can differ greatly between DL models trained on the same data. A method for simulating image data that can then be used for UQ quality assessment is described. A general method for simulating realistic data for the purpose of assessing a model’s UQ quality is also presented. A Bayesian uncertainty framework for understanding uncertainty and existing metrics is described. Research that came out of collaborations with two university partners are discussed along with a software toolkit that is currently being developed to implement the UQ quality assessment framework as well as serve as a general guide to incorporating UQ into DL applications.
97 MATHEMATICS AND COMPUTING↗
Applications of Federated Learning in Semiconductor Manufacturing [Poster]
As semiconductor manufacturers explore advanced data analytics and modeling techniques and data hungry machine learning models increase in popularity due to their accuracy in solving generalized problems and ability to learn complex relationships, federated learning emerges as a privacy preserving machine learning technique for preserving data privacy and ensuring intellectual property protection. Federated Learning is a machine learning technique focused on training models using distributed data that never needs to be centrally stored, allowing the use of advanced machine learning techniques without compromising data privacy, and in the semiconductor manufacturing industry advanced machine learning techniques can reduce cost and time, but maintaining data privacy is essential to maintaining a competitive advantage. This paper systematically reviews existing literature on applications of federated learning in the semiconductor manufacturing industry with a focus on identifying common themes, algorithms, and gaps within the literature to drive future research directions. The findings reveal five key themes, including improvements in quality assurance, virtual models, privacy preservation, reliable data practices, and emerging trends and developments. By identifying key themes in literature on federated learning and semiconductor manufacturing and analyzing gaps and discussed methodologies, this study highlights several potential future research directions to expand the application of federated learning techniques in the semiconductor manufacturing domain.
42 ENGINEERING↗
Robust inference of the Galactic Centre gamma-ray excess spatial properties
ABSTRACT The gamma-ray Fermi-LAT Galactic Centre excess (GCE) has puzzled scientists for over 15 yr. Despite ongoing debates about its properties, and especially its spatial distribution, its nature remains elusive. We scrutinize how the estimated spatial morphology of this excess depends on models for the Galactic diffuse emission, focusing particularly on the extent to which the Galactic plane and point sources are masked. Our main aim is to compare a spherically symmetric morphology – potentially arising from the annihilation of dark matter (DM) particles – with a boxy morphology – expected if faint unresolved sources in the Galactic bulge dominate the excess emission. Recent claims favouring a DM-motivated template for the GCE are shown to rely on a specific Galactic bulge template, which performs worse than other templates for the Galactic bulge. We find that a non-parametric model of the Galactic bulge derived from the VISTA Variables in the Via Lactea survey results in a significantly better fit for the GCE than DM-motivated templates. This result is independent of whether a galprop-based model or a more non-parametric ring-based model is used to describe the diffuse Galactic emission. This conclusion remains true even when additional freedom is added in the background models, allowing for non-parametric modulation of the model components and substantially improving the fit quality. When adopted, optimized background models provide robust results in terms of preference for a boxy bulge morphology for the GCE, regardless of the mask applied to the Galactic plane.
Astronomy & Astrophysics↗
Effects of Rapid Permafrost Thaw on CO2 and CH4 Fluxes in a Warmer and Wetter Future (Final Technical Report)
When ice-rich permafrost thaws, the ground subsides, creating thermokarst landscapes with dramatically different soil conditions and carbon fluxes than the original ecosystem. Roughly 20% of the northern permafrost region is susceptible to thermokarst formation (Olefeldt et al., 2016). Thermokarst formation is often rapid; tens of meters of permafrost can thaw within a few years (Schuur et al., 2015). In topographically low areas, thermokarst thaw converts boreal forest or tundra dry shrub ecosystems into sedge or Sphagnum moss wetlands (Olefeldt et al., 2016). While this type of landscape transformation releases carbon stored in permafrost into the atmosphere, on longer time scales, it facilitates sequestration of atmospheric carbon in plant biomass because permafrost thaw releases plant-available nutrients and wetlands are highly productive (M. C. Jones et al., 2017). However, wetlands also generate methane, which is a potent greenhouse gas. Methane emissions from thermokarst wetlands can cause these carbon-sequestering systems to have a positive global warming potential (Johansson et al., 2006; Turetsky et al., 2007). Our project objective was to improve Earth System and environmental predictability by advancing understanding of how CO 2 and CH 4 flux in permafrost thaw-induced wetlands (thermokarst) will change in the future as temperatures and climate conditions shift. Northern latitudes are expected to get warmer and wetter (IPCC 2013), and initiation and expansion of thermokarst thaw is expected to increase (Jorgenson et al. 2006; Zhang et al. 2017). Given these expected changes, our work sought to address three broad questions: Q1) How will northern latitude CO2 and CH4 emissions respond to warming temperatures? Q2) What is the impact of precipitation on permafrost thaw and carbon emissions? Q3) How do CO2 and CH4 emissions change as wetlands age after permafrost thaw? To answer these questions, we took both a modeling and measurement approach. Modeling work was conducted with DOE’s Energy Exascale Earth System Model (E3SM) land model (ELM). Empirical work took place in two primary locations. The first was a thermokarst site near Fairbanks, AK. The site is part of the Bonanza Creek Long Term Ecological Research program and is well instrumented (Neumann et al., 2019). The second was an isolated thawing permafrost wetland located on Kenai Peninsula — Brown’s Lake bog (B. M. Jones et al., 2016) — where the current climate is representative of what is expected at higher latitudes in the future. This site provides an ideal opportunity to test our hypotheses about how thermokarst wetland dynamics will respond to future environmental conditions. In addition, the project collaborated with researchers asking similar questions who were collecting measurements in high latitude post-glacial lakes located in collaborators tackling similar questions in Sweden (Stordalen Mire) (Emerson et al., 2021). Project efforts directly aligned with the stated goal of the funding opportunity announcement (FOA), which was “to improve the understanding and representation of terrestrial ecosystems in ways that advance Earth system model parameterizations and capabilities... thereby improving the quality of Earth and environmental model projections and providing the scientific foundation needed to support DOE’s science and energy missions.” Specifically, the project improved sophistication and accuracy of the Energy Exascale Earth System Model (E3SM), which is being developed primarily at DOE National Laboratories to support scientific research and decision-making.
54 ENVIRONMENTAL SCIENCES↗
AI Model Benchmarking for Nonproliferation Applications: Steel Thread Benchmarking Task Force Technical Report (Rev. 2)
Steel Thread is a NA-22 venture that seeks to build trustworthy, reliable AI models that can be used in a wide variety of nonproliferation tasks. A key aspect of building these models is developing appropriate benchmarks and evaluation methods, which will enable the venture to identify and adapt models to provide the most value in the nonproliferation domain. Benchmarks must be relevant to key tasks in this domain, such as question answering, information retrieval, document summarization and classification, consensus analysis, and image and data analysis. This report 1) provides an overview of benchmark design, evaluation, and challenges; 2) reviews a variety of open benchmarks, with a focus on language models and tasks; and 3) identifies benchmarks that are most relevant to Steel Thread. This report is intended to serve as a basis for further efforts to classify and evaluate benchmarks and their correlation with success on nonproliferation-specific tasks. The Steel Thread venture has defined benchmarks to be a particular combination of a dataset (or datasets) and a metric (or metrics) conceptualized as representing one or more specific tasks or sets of abilities for a specific modality. It is adopted by a research community as a shared framework for comparing methods.1 It includes 1) Data: Labeled (a designated subset not used for training, which could be all the data), 2) Metric: A way to quantify performance, 3) Task/Ability: What the benchmark is testing, 4) Protocol: A structured and repeatable evaluation process, 5) Baseline/Reference Model: For comparison; could be statistical, rule-based, SME-derived, or another model, and 6) Maintenance Plan: to update with new information over time; important for long-term utility. For further clarity, the definition includes what a benchmark, in this context, is not. It is not a corpus of training data, specific to a model (it is intended to apply to a range of models), a universal evaluation of performance, a guarantee that the ‘top’ model on the leaderboard will be the best fit for every specific use case, an all-encompassing proof of a model’s universal quality, nor is it a one-size-fits-all measure of success. It does not cover every real-world constraint (like operational, ethical, or cost considerations), a systems integration test, or a unit test. This definition was inspired by and resulted from discussions within the Steel Thread Benchmarking Task Force. This group was formed to define what we would mean as a benchmark within Steel Thread but persisted as the need to develop a thorough understanding of the large and expanding existing benchmarking space. This technical report is a result of the group’s divide and conquer approach to exploring this space. The release of benchmarks might not be progressing as quickly as model development, but it is moving very fast, as many benchmarks quickly become saturated, when state-of-the-art models score so close to the benchmark’s ceiling that their results are virtually indistinguishable. At that point, the test no longer differentiates between new systems, so researchers usually stop reporting scores as the benchmark no longer informs about improvements from the next generation of models. In the OpenAI announcement of GPT-5, they reported results on six flagship public benchmarks (AIME 2025, SWE-bench Verified, Aider Polyglot, MMMU, HealthBench Hard, GPQA) but the full system-card covers roughly thirty-five separate evaluations, comprising hundreds of test task items in total. There have been some efforts to summarize benchmarks in specific fields, like for text-to-image generation, but these surveys have had a narrow methodology scope. Therefore, a comprehensive survey of all benchmarks or even all benchmarks that could be relevant to Steel Thread is outside of the scope of this report. We chose some specific benchmarks to investigate in detail.
97 MATHEMATICS AND COMPUTING↗
Analytic error analysis of cross section interpolation methods in nodal diffusion codes - I : Theory
This paper discusses two cross section interpolation methods commonly found in popular nodal codes; the partial derivatives and multiple tables models. The motivation for choosing a model, and thus a case matrix structure, is a trade off between accuracy and computational cost. Due to decades of experience, there are default structures that are sufficient for current light water reactor analysis. However, this is not necessarily the case for advanced reactor designs. Therefore, it is advantageous to understand the sources of error in cross section interpolation models so that the quality of a case matrix may be improved. A mathematical framework for these models is presented in this work that provides a more rigorous connection between the nuclear engineering field's cross section interpolation methods and the broader mathematical field of function approximation. The two cross section models examined in this paper were found to utilize Lagrange interpolation and are a subset of Lagrange tensor products. Classical results of Lagrange polynomial error analysis were then applied to the partial derivative and multiple tables models to derive expressions for the total point-wise error. The analytical results classify the total error into two parts: the model form error and interpolation error. Finally, based on our observations, a better foundation for improving the quality of a case matrix is proposed. (authors)
22 GENERAL STUDIES OF NUCLEAR REACTORS↗
Covariance-Free Bifidelity Control Variates Importance Sampling for Rare Event Reliability Analysis
Multifidelity modeling has been steadily gaining attention as a tool to address the problem of exorbitant model evaluation costs that makes the estimation of failure probabilities a significant computational challenge for complex real-world problems, particularly when failure is a rare event. To implement multifidelity modeling, estimators that efficiently combine information from multiple models/sources are necessary. In past works, the variance reduction techniques of control variates (CV) and importance sampling (IS) have been leveraged for this task. In this paper, we present the CVIS framework—a creative take on a coupled CV and IS estimator for bifidelity reliability analysis. The framework addresses some of the practical challenges of the CV method by using an estimator for the control variate mean and sidestepping the need to estimate the covariance between the original estimator and the control variate through a clever choice for the tuning constant. Furthermore, the task of selecting an efficient IS distribution is also considered, with a view towards maximally leveraging the bifidelity structure and maintaining expressivity. Additionally, a diagnostic is provided that indicates both the efficiency of the algorithm as well as the relative predictive quality of the models utilized. Finally, the behavior and performance of the framework is explored through analytical and numerical examples.
Markov chain Monte Carlo↗
Atomic protein structure refinement using all-atom graph representations and SE(3)-equivariant graph transformer
Abstract Motivation The state-of-art protein structure prediction methods such as AlphaFold are being widely used to predict structures of uncharacterized proteins in biomedical research. There is a significant need to further improve the quality and nativeness of the predicted structures to enhance their usability. In this work, we develop ATOMRefine, a deep learning-based, end-to-end, all-atom protein structural model refinement method. It uses a SE(3)-equivariant graph transformer network to directly refine protein atomic coordinates in a predicted tertiary structure represented as a molecular graph. Results The method is first trained and tested on the structural models in AlphaFoldDB whose experimental structures are known, and then blindly tested on 69 CASP14 regular targets and 7 CASP14 refinement targets. ATOMRefine improves the quality of both backbone atoms and all-atom conformation of the initial structural models generated by AlphaFold. It also performs better than two state-of-the-art refinement methods in multiple evaluation metrics including an all-atom model quality score—the MolProbity score based on the analysis of all-atom contacts, bond length, atom clashes, torsion angles, and side-chain rotamers. As ATOMRefine can refine a protein structure quickly, it provides a viable, fast solution for improving protein geometry and fixing structural errors of predicted structures through direct coordinate refinement. Availability and implementation The source code of ATOMRefine is available in the GitHub repository (https://github.com/BioinfoMachineLearning/ATOMRefine). All the required data for training and testing are available at https://doi.org/10.5281/zenodo.6944368.
59 BASIC BIOLOGICAL SCIENCES↗
MIXv2: a long-term mosaic emission inventory for Asia (2010–2017)
The MIXv2 Asian emission inventory is developed under the framework of the Model Inter-Comparison Study for Asia (MICS-Asia) Phase IV and produced from a mosaic of up-to-date regional emission inventories. We estimated the emissions for anthropogenic and biomass burning sources covering 23 countries and regions in East, Southeast and South Asia and aggregated emissions to a uniform spatial and temporal resolution for seven sectors: power, industry, residential, transportation, agriculture, open biomass burning and shipping. Compared to MIXv1, we extended the dataset to 2010–2017, included emissions of open biomass burning and shipping, and provided model-ready emissions of SAPRC99, SAPRC07, and CB05. A series of unit-based point source information was incorporated covering power plants in China and India. A consistent speciation framework for non-methane volatile organic compounds (NMVOCs) was applied to develop emissions by three chemical mechanisms. The total Asian emissions for anthropogenic/open biomass sectors in 2017 are estimated as follows: 41.6/1.1 Tg NO x , 33.2/0.1 Tg SO 2 , 258.2/20.6 Tg CO, 61.8/8.2 Tg NMVOC, 28.3/0.3 Tg NH 3 , 24.0/2.6 Tg PM 10 , 16.7/2.0 Tg PM 2.5 , 2.7/0.1 Tg BC (black carbon), 5.3/0.9 Tg OC (organic carbon), and 18.0/0.4 Pg CO 2 . The contributions of India and Southeast Asia were emerging in Asia during 2010–2017, especially for SO 2 , NH 3 and particulate matter. Gridded emissions at a spatial resolution of 0.1° with monthly variations are now publicly available. This updated long-term emission mosaic inventory is ready to facilitate air quality and climate model simulations, as well as policymaking and associated analyses.
54 ENVIRONMENTAL SCIENCES↗
Increased nitrogen use efficiency in crop production can provide economic and environmental benefits
Potential economic and environmental benefits of increasing nitrogen-use efficiency (NUE) are widely recognized but scarcely quantified. This study quantifies the effects of increased NUE on 1) the national agricultural economy using a simulation model of US agriculture and 2) regional water quality effects using a biogeochemical model for the Arkansas-White-Red river basin. Here, national economic effects are reported for NUE improvement scenarios of 10%, 20%, 50%, and 100%, whereas regional water quality effects are estimated for a 20% NUE improvement scenario in the Arkansas-White-Red river basin. Simulating a 20% increase in NUE in row crops is shown to reduce N requirements by 1.4 million tonnes y-1 and increase farmer net profits by 1.6% ($743 million) per year by 2026 over the baseline simulation for the same period. For each 10% increase in NUE, annual farm revenues for commodity crops increased over the baseline by approximately $350 million per year by 2026. Changes in crop prices and land-use relative to the baseline were less than 2%. This suggests a net benefit even though fertilizer cost savings can result in increased cultivation of land, i.e., ‘Jevon's paradox’. Results from the biogeochemical model of the Arkansas-White-Red river basin suggest that a 20% increase in NUE corresponds to a 5.72% reduction in nitrate loadings to freshwaters, with higher reductions in agricultural watersheds. The value of these reductions was estimated as $43 ha-1, for a total of $15.3 to 136.7 million yr-1 in avoided water treatment costs. After estimating the social value of increased NUE, we conclude with a discussion of potential strategies to increase efficiency and the research needed to achieve this goal. These include perennialization of the agricultural landscape, genetic crop improvement, targeted fertilizer application, and manipulation of the plant-root microbiome.
54 ENVIRONMENTAL SCIENCES↗
Metrics as tools for bridging climate science and applications
In climate science and applications, the term “metric” is used to describe the distillation of complex, multifaceted evaluations to summarize the overall quality of a model simulation, or other data product, and/or as a means to quantify some response to climate change. Metrics provide insights into the fidelity of processes and outcomes from climate models and can assist with both differentiating models' representation of variables or processes and informing whether models are “fit for purpose.” Metrics can also provide a valuable reference point for co-production of knowledge between climate scientists and climate impact practitioners. Although continued metric developments enable model developers to better understand the impacts of decisions made in the model design process, metrics also have implications for the characterization of uncertainty and facilitating analyses of underlying physical processes. As a result, comprehensive evaluation with multiple metrics enhances usability of climate information by both scientific and stakeholder communities. Here, this paper presents examples of insights gained from the development and appropriate use of metrics, and provides examples of how metrics can be used to engage with stakeholders and inform decision-making.
54 ENVIRONMENTAL SCIENCES↗