Results for “data quality metrics” — Page 14

DOE OSTI2022

Production Agency Quality Assurance Management Execution Strategy [Thesis]

There are multiple federal directives that Los Alamos National Laboratory (LANL) must follow for the proper implementation of quality assurance, strategic planning, and execution of manufacturing and surveillance operations. Recent assessments identified that manufacturing in all processing areas is dynamic due to influencing scope changes, design modifications, funding adjustments, and staff attrition. In response, this project was started to develop a comprehensive PAQ execution strategy to support the success of LANL’s manufacturing mission. This project’s method for developing an updated architecture was to create an integrated, flexible, reliable, and agile strategic process. The execution plan has five deliverables: 1) an integrated schedule view of PAQ work scope, 2) a resource management plan that aligns with the required scope, 3) a metrics monitoring dashboard, 4) a project management plan for sustaining NAP 401.1A implementation, and 5) a risk management plan. The research design is a mixed-method approach focused on data collection for each deliverable. The methodology entails qualitative and quantitative research, metrics monitoring, data analysis, and the creation of a dashboard. The qualitative methods used include literature reviews and interviews. The quantitative methods include resource, financial, schedule and survey data analysis. Preliminary and final results were peer reviewed by subject matter experts both within the ALDWP organization and deployed support.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2024

Interlaced Characterization and Calibration (ICC) for Improved Computational Simulation Credibility

Accurate material characterization and model calibration are pivotal for simulations used for high-consequence engineering decisions. Current characterization and calibration methods (1) use simplified test specimen geometries and global data, (2) cannot guarantee that sufficient characterization data is collected for a specific model of interest, (3) provide only mean parameter values with no uncertainty quantification, and (4) are sequential, inflexible, and time-consuming. This work developed a new paradigm—coined Interlaced Characterization and Calibration (ICC)—which drives forward the state-of-the-art in model calibration by bringing together recent advancements into one improved workflow. The ICC paradigm (1) employs tools to efficiently use full-field data to calibrate high-fidelity material models, (2) aligns the data needed with the data collected by adopting an optimal experimental design protocol, (3) provides uncertainty metrics on the calibrated model parameters, and (4) incorporates these advances into a quasi real-time feedback loop. The ICC framework was validated synthetically with both low-fidelity and high-fidelity simulations paired with several different elastoplastic material models, and was also demonstrated experimentally with an aluminum 6061 cruciform exemplar specimen. Results showed that the ICC framework—in which Bayesian optimal experimental design actively guided the experiment— resulted in calibrations with similar or better accuracy than predetermined experiments based on subject matter expertise. Moreover, the ICC framework produced a complete model calibration— with quantified uncertainties on model parameters—in 1 week, a 5 - 10× increase in efficiency over traditional approaches. Thus, the ICC paradigm improves both the calibration process and quality, by (1) improving efficiency, which increases agility of solid mechanics modeling and enables utilization of computational simulation (CompSim) at earlier stages of the design cycle and (2) providing quantified, and in some cases reduced, parameter uncertainties, which increases confidence in model predictions and supports credible decision making.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

The Dynamic Networks Experiments: Virtual Experiments to Quantify Gains in Nuclear Explosion Monitoring

We describe an ongoing series of virtual experiments conducted collaboratively by four United States National Laboratories: Sandia National Laboratories, Los Alamos National Laboratory, Lawrence Livermore National Laboratory, and Pacific Northwest National Laboratory. These Dynamic Network Experiments (DNEs) provide an experimental framework to evaluate the potential impact of new research tools on nuclear explosion monitoring. The second DNE (DNE2), completed in 2024, exploited waveform data (seismic, infrasound, and electromagnetic) that was recorded by multi-modal sensors within and near the Nevada National Security Site and synthetic radionuclide signatures over multiple time periods. During the execution of DNE2, we processed and analyzed data through a multi-stage event processing pipeline that ingested raw data, performed quality control, detected signals, built events from these signals, located these events, and characterized the events’ source types and sizes. For each stage and over the entire event processing pipeline, we evaluated performance changes by comparing the performance of new data processing methods, models, and algorithms against a baseline. We also performed an additional execution phase to assess event processing pipeline function, speed, and efficiency against that of an expert analyst, including computational and manual efforts. Finally, we assessed the impact and effort of modern computing infrastructure on the monitoring pipeline. This paper describes key elements of the DNEs, from formulation through execution, as demonstrated in DNE2. The DNEs introduce several novel concepts to quantitatively measure the potential impact of new methods on explosion monitoring, including the collaborative design of multi-modal datasets, performance and logistical metrics, and integrated analyses.

42 ENGINEERING↗

DOE OSTI2020

An international laboratory comparison of dissolved organic matter composition by high resolution mass spectrometry: Are we getting the same answer?

Since the first detailed compositional analysis of dissolved organic matter (DOM) in 1995, high resolution mass spectrometry (HRMS) has become a vital tool for DOM characterization. While the upward trend in HRMS for molecular level analysis of DOM continues, so do the challenges of data comparison and interpretation between laboratories operating instruments of differing performance and user operating conditions. It is therefore essential that the community establishes whether data and trends can be compared robustly between research groups. To this end, 4 identically prepared DOM samples were each studied by 16 laboratories, representing 17 commercially-purchased instruments located across 8 countries, using positive- and negative-mode electrospray ionization (ESI) HRMS analysis. Despite the widely reported sensitivity of ESI experiments to sample matrix, preparation, ion source operation and instrument tuning parameters, the instruments used in this study showed relatively low variability between results for the same samples, particularly H/C and AI metrics. The variation in O/C and m/z metrics was relatively higher, although z-score graphs and %BCD demonstrated that the relative difference in general chemical composition between samples was the same for each instrument included in the study. As such, these metrics can be used for fingerprinting of DOM samples and as a benchmark for quality control in participating laboratories, with future strides in environmental science building upon these efforts.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2023

Algal Biomass Production via Open Pond Algae Farm Cultivation: 2022 State of Technology and Future Research

The annual State of Technology (SOT) assessment is an essential activity for platform research conducted under the Bioenergy Technologies Office (BETO). It allows for the impact of research progress (both directly achieved in-house at the National Renewable Energy Laboratory [NREL] and furnished by partner organizations) to be quantified in terms of economic improvements in the overall biofuel production process for a particular biomass processing pathway, whether based on terrestrial or algal biomass feedstocks. As such, initial benchmarks can be established for currently demonstrated performance, and progress can be tracked toward out-year goals to ultimately demonstrate economically viable biofuel technologies. NREL's algae SOT benchmarking efforts focus both on front-end algal biomass production and separately on back-end conversion to fuels through NREL's "combined algae processing" (CAP) pathway. The production model is based on outdoor long-term cultivation data, enabled by comprehensive algal biomass production trials conducted under the Development of Integrated Screening, Cultivar Optimization, and Verification Research (DISCOVR) consortium efforts, driven by data furnished by Arizona State University (ASU) at the Arizona Center for Algae Technology and Innovation (AzCATI) testbed site. The CAP model is based on experimental efforts conducted primarily under NREL research and development projects. This report focuses on front-end algal biomass production, documenting the pertinent algal biomass cultivation parameters that were input to the NREL open pond algae farm model. Through partnerships under DISCOVR, collaborators at ASU furnished details on cultivation performance metrics including biomass productivity and harvest densities for recent growth trials done at the AzCATI site. The resulting biomass productivity was calculated at 18.5 g/m2/day (ash-free dry weight [AFDW], annual average) for seasonal cultivation of Picochlorum celeri, Tetraselmis striata LANL1001, and Monoraphidium minutum 26B-AM biomass strains at the ASU site. Picochlorum celeri achieved the best productivity from May to September, with Monoraphidium minutum 26B-AM being used in October, November, March, and April, and Tetraselmis striata employed during winter months (December through February). Beyond the standard SOT models, in Appendix C of this report we also present an industry case study evaluating several scenarios reflective of outdoor cultivation data furnished by an industry collaborator. This case study provides a supplementary datapoint on work being performed elsewhere achieving comparable cultivation productivity with more favorable compositional quality, producing biomass enriched in lipids as may be more optimal for conversion upgrading to fuels and products.

09 BIOMASS FUELS↗

DOE OSTI2024

Assessing Resilience in Lane Detection Methods: Infrastructure-Based Sensors and Traditional Approaches for Autonomous Vehicles

Traditional autonomous vehicle perception subsystems that use onboard sensors have the drawbacks of high computational load and data duplication. Infrastructure-based sensors, which can provide high quality information without the computational burden and data duplication, are an alternative to traditional autonomous vehicle perception subsystems. However, these technologies are still in the early stages of development and have not been extensively evaluated for lane detection system performance. Therefore, there is a lack of quantitative data on their performance relative to traditional perception methods, especially during hazardous scenarios, such as lane line occlusion, sensor failure, and environmental obstructions. We address this need by evaluating the influence of hazards on the resilience of three different lane detection methods in simulation: (1) traditional camera detection using a U-Net algorithm, (2) radar detections using infrastructure-based radar retro-reflectors (RRs), and (3) direct communication of lane line information using chip-enabled raised pavement markers (CERPMs). The performance of each of these methods is assessed using resilience engineering metrics by simulating the individual methods for each sensor technology’s response to related hazards in the CARLA simulator. Using simulation techniques to replicate these methods and hazards acquires extensive datasets without lengthy time investments. Specifically, the resilience triangle was used to quantitatively measure the resilience of the lane detection system to obtain unique insights into each of the three lane detection methods; notably the infrastructure-based CERPMs and RRs had high resistance to hazards and were not as easily affected as the vision-based U-Net. However, while U-Net was able to recover the fastest from the disruption as compared to the other two methods, it also had the most performance loss. Overall, this study demonstrates that while infrastructure-based lane keeping technologies are still in early development, they have great potential as alternatives to traditional ones.

Patil, Pritesh↗

DOE OSTI2022

Marine Renewable Energy Applications for Restorative Ocean Farming: Kelp

Kelp farming and kelp forest restoration have both been proposed as a solution to locally decrease the impacts of ocean acidification and eutrophication, often with co-benefits to other forms of aquaculture and mariculture. Compared to global markets, the kelp industry in the United States is still in its early phases, with the first commercial kelp farm founded in Casco Bay, Maine in 2010. Since then, interest and effort in kelp production has been increasing, with farms now present in Maine, New Hampshire, Connecticut, Rhode Island, Massachusetts, New York, Washington, and Alaska. Many research projects are underway in the United States to explore benefits of 3D ocean farming, tackle logistical problems of working in the ocean, autonomous farming, and explore viable end uses for kelp products. In seaweed farming, to remove the stored carbon or excess nutrients from the system, the biomass needs to be harvested at the optimal time to avoid the release of CO 2 that comes with decomposition. Timing of the harvest is also important for maximum crop yield, which can vary based on the final product. Additional monitoring needs can include a variety of water quality metrics, growth measurements, and visuals to ensure the health of the farm, comply with permits, support operations and maintenance functions. The variables measured may vary by desired end use of the product, location of farm, and operational design. Monitoring all of these parameters requires specialized devices that can be costly and challenging to maintain. Monitoring devices often face power and logistical constraints that could prevent kelp farmers from adopting these technologies or receiving accurate, efficient monitoring to assess ecosystem benefits and valuation. Marine energy has been identified as a possible power source for these devices. This project investigates the power needs for conducting kelp farm environmental monitoring compared with the available marine energy resource to evaluate if locally generated ocean energy could provide a solution to these monitoring challenges and benefit kelp farmers. This process was structured as follows: 1. Define what data is needed for farmers and their communities through desk research and interviews with end users. 2. Identify sensors and power requirements currently in use or available for commercial purchase. 3. Analyze current kelp and other mariculture farm locations for the potential marine energy resource. 4. Analyze farm designs and associated structures to make recommendations for marine energy design. 5. Quantify value that investment in sensors could provide in terms of carbon credit possibilities.

09 BIOMASS FUELS↗

DOE OSTI2021

Assessing data change in scientific datasets

Summary Scientific datasets are growing rapidly and becoming critical to next‐generation scientific discoveries. The validity of scientific results relies on the quality of data used and data are often subject to change, for example, due to observation additions, quality assessments, or processing software updates. The effects of data change are not well understood and difficult to predict. Datasets are often repeatedly updated and recomputing derived data products quickly becomes time consuming and resource intensive and may in some cases not even be necessary, thus delaying scientific advance. Despite its importance, there is a lack of systematic approaches for best comparing data versions to quantify the changes, and ad‐hoc or manual processes are commonly used. In this article, we propose a novel hierarchical approach for analyzing data changes, including real‐time (online) and offline analyses. We employ a variety of fast‐to‐compute numerical analyses, graphical data change representations, and more resource‐intensive recomputations of a subset of the data product. We illustrate the application of our approach using three scientific diverse use cases, namely, satellite, cosmological, and x‐ray data. The results show that a variety of data change metrics should be employed to enable a comprehensive representation and qualitative evaluation of data changes.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Quantifying Solar PV Variability at Multiple Timescales for Power Systems Studies: Preprint

Increasing penetration of solar photovoltaics (PV) has concerned power system operators about their ability to balance load and generation over multiple timescales and maintain acceptable frequency profile because of the variability and uncertainty associated with solar PV. This paper focuses on developing metrics to quantify solar PV variability at multiple temporal scales ranging from seconds to an hour to help power system operators and planners better understand and account for solar PV variability under high solar PV penetration. Fourier coefficients of solar irradiation spectrum and average size of solar irradiation ramps over a day are the two metrics that are used to quantity the variability. It is shown that by using the two metrics the days with highest and lowest solar irradiation variability can be identified, which is useful information for operations planning or long-term power system planning studies. The key findings of the paper are (i) single sensor-based variability quantification can significantly exaggerate the variability of a solar PV plant, particularly at seconds to 1-min temporal resolution, (ii) at 1-minute or lower temporal resolutions, single sensor-based variability quantification of an entire plant can be fairly accurate, with the accuracy increasing as the temporal resolution reduces from 1-minute to 1-hour, (iii) at each 1-minute and lower temporal resolutions, maximum solar irradiation ramps appear to be independent of the geographical location of the sensor, and (iv) significant reduction in solar PV plant variability over a region with multiple plants is possible at all temporal scales, but more high quality, high temporal resolution data is needed to quantify the reduction with high confidence.

photovoltaics↗

DOE OSTI2025

Using a Large Language Model as a Building Block to Generate Usable Validation and Verification Suite for OpenMP

In the HPC area, both hardware and software move quickly. Often new hardware is developed and deployed, the corresponding software stack, including compilers and other tools, are under active development while leading edge software developers are working to port and tune their applications, all at the same time. While the software ecosystem is in flux, one of the key challenges for users is obtaining insight into the state of implementation of key features in the programming languages and models their applications are using – whether they have been implemented, and whether the implementation conforms to the specification, especially for newly implemented features (less tested by widespread use). OpenMP is one of the most prominent shared memory programming models used for on-node programming in HPC. With the shift towards accelerators (such as GPUs and FPGAs) and heterogeneous programming OpenMP features are getting more complex. It is natural to ask whether generative AI approaches, and large language models (LLMs) in particular, can help in producing validation and verification test suites to allow users better and faster insights into the availability and correctness of OpenMP features of interest. In this work, we explore the use of ChatGPT-4 to generate a suite of tests for OpenMP features. We have chosen a set of directives and clauses, a total of 78 combinations, which first appeared in OpenMP 3.0 (released in May 2008) but are also relevant for accelerators. We prompted ChatGPT to generate tests in the C and Fortran languages, for both host (CPU) and device (accelerator). On the Summit super-computer using the GNU implementation, we found that, of the 78 generated tests 67 C tests and 43 Fortran tests compiled successfully and fewer than those executed to completion. On further analysis we show that not all generated tests are valid. We document the process, results, and provide detailed analysis regarding the quality of tests generated. With the aim of providing input to a production quality validation and verification suite, we manually implement the corrections required to make the tests valid according to the current OpenMP specification. We quantify this effort as small, medium, or large, and record the lines of code changed to correct the invalid tests. With the corrected tests we validate recent implementations from HPE, AMD, and GNU on the Frontier supercomputer. Our experiment and subsequent analysis show that although LLMs are capable of producing HPC specific codes, they are limited by their understanding of the deeper semantics and restrictions of programming models such as OpenMP. Unsurprisingly more commonly used features have better support, while some OpenMP 3.0 directives such as sections and tasking are not universally supported on accelerators. We demonstrate that successful compilation and execution to completion are inadequate metrics for evaluating generated code and that, at this time, commodity LLMs require expert intervention for code verification. This points to gaps in the training data that is currently available for HPC. We demonstrate that with "small" effort 37% of generated invalid C tests and 63% of generated invalid Fortran tests could be corrected. This improves productivity of test generation as we circumvent writing from scratch and the common programming errors associated with it.

Pophale, Swaroop [ORNL] (ORCID:0000000185446367)↗

DOE OSTI2023

Enhanced Surgical Decision-Making Tools in Breast Cancer: Predicting 2-Year Postoperative Physical, Sexual, and Psychosocial Well-Being following Mastectomy and Breast Reconstruction (INSPiRED 004)

Background: We sought to predict clinically meaningful changes in physical, sexual, and psychosocial well-being for women undergoing cancer-related mastectomy and breast reconstruction 2 years after surgery using machine learning (ML) algorithms trained on clinical and patient-reported outcomes data. Patients and Methods: We used data from women undergoing mastectomy and reconstruction at 11 study sites in North America to develop three distinct ML models. We used data of ten sites to predict clinically meaningful improvement or worsening by comparing pre-surgical scores with 2 year follow-up data measured by validated Breast-Q domains. We employed ten-fold cross-validation to train and test the algorithms, and then externally validated them using the 11th site’s data. We considered area-under-the-receiver-operating-characteristics-curve (AUC) as the primary metric to evaluate performance. Results: Overall, between 1454 and 1538 patients completed 2 year follow-up with data for physical, sexual, and psychosocial well-being. In the hold-out validation set, our ML algorithms were able to predict clinically significant changes in physical well-being (chest and upper body) (worsened: AUC range 0.69–0.70; improved: AUC range 0.81–0.82), sexual well-being (worsened: AUC range 0.76–0.77; improved: AUC range 0.74–0.76), and psychosocial well-being (worsened: AUC range 0.64–0.66; improved: AUC range 0.66–0.66). Baseline patient-reported outcome (PRO) variables showed the largest influence on model predictions. Conclusions: Machine learning can predict long-term individual PROs of patients undergoing postmastectomy breast reconstruction with acceptable accuracy. This may better help patients and clinicians make informed decisions regarding expected long-term effect of treatment, facilitate patient-centered care, and ultimately improve postoperative health-related quality of life.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2024

Algal Biomass Production via Open Pond Algae Farm Cultivation: 2023 State of Technology and Future Research

The annual State of Technology (SOT) assessment is an essential activity for platform research conducted under the Bioenergy Technologies Office (BETO). It allows for the impact of research progress (both directly achieved in-house at the National Renewable Energy Laboratory [NREL] and furnished by partner organizations) to be quantified in terms of economic improvements in the overall biofuel production process for a particular biomass processing pathway, whether based on terrestrial or algal biomass feedstocks. As such, initial benchmarks can be established for currently demonstrated performance, and progress can be tracked toward out-year goals to ultimately demonstrate economically viable biofuel technologies. NREL's algae SOT benchmarking efforts historically focused both on front-end algal biomass production and separately on back-end conversion to fuels through NREL's "combined algae processing" (CAP) pathway. The production model is based on outdoor long-term cultivation data, enabled by comprehensive algal biomass production trials conducted under the Development of Integrated Screening, Cultivar Optimization, and Verification Research (DISCOVR) consortium efforts, driven by data furnished by Arizona State University (ASU) at the Arizona Center for Algae Technology and Innovation (AzCATI) testbed site. The CAP model is based on experimental efforts conducted primarily under NREL research and development projects. This report focuses on front-end algal biomass production, documenting the pertinent algal biomass cultivation parameters that were input to the NREL open pond algae farm model. Through partnerships under DISCOVR, collaborators at ASU furnished details on cultivation performance metrics including biomass productivity and harvest densities for recent growth trials done at the AzCATI site. The resulting biomass productivity was calculated at 16.7 g/m 2 /day (ash-free dry weight [AFDW], annual average) for seasonal cultivation of Picochlorum celeri TG2 and Monoraphidium minutum 26B-AM biomass strains at the ASU site. Picochlorum celeri achieved the best productivity from April to September, with Monoraphidium minutum 26B-AM being used between October and March. Tetraselmis striata LANL1001, usually part of the strain rotation in previous cultivation SOTs, was supplanted by Monoraphidium minutum 26B-AM in this year's outdoor cultivation trials. Finally, building from an industry case study presented in the 2022 SOT report, in the Appendix of this report we provide an update on further improved data furnished by an industry collaborator and resultant impacts on economics reflecting several seasonal scenarios. This case study provides a supplementary datapoint on work being performed elsewhere with a more dedicated focus on improved compositional quality, producing biomass enriched in lipids as may be more optimal for conversion upgrading to fuels and products.

09 BIOMASS FUELS↗

DOE OSTI2021

JGI QC impact on assembly, binning, phylogenomics, and functional analysis

Background Investigators using metagenomic sequencing to study their microbiomes are often provided data that has been trimmed and decontaminated or do it themselves without knowing the effect these procedures can have on their downstream analyses. Here we evaluated the impact that JGI trimming and decontamination procedures had on assembly and binning metrics, placement of metagenome assembled genomes into species trees, and functional profiles of metagenome-assembled genomes (MAGs) extracted from twenty three complex rhizosphere metagenomes. We also investigated how more aggressive trimming impacts these binning metrics. Results We found that JGI trimmed and decontamination of input reads had some significant impacts in assembly and binning metrics compared to raw reads, and that differences in placement of MAGs in species trees increased with decreasing completeness and contamination thresholds. More aggressive trimming beyond those used by JGI were found to reduce MAG counts. Conclusions Mild trimming and decontamination of metagenomics reads prior to assembly can change an investigator’s answer to the questions, “Who is there and what are they doing? However, mild trimming and decontamination of metagenomic reads with high quality scores is recommended for those who elect to do so.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2021

Impact of BBDuk metagenomic read trimming and decontamination

Background Investigators using metagenomic sequencing to study their microbiomes are often provided data that has been trimmed and decontaminated or do it themselves without knowing the effect these procedures can have on their downstream analyses. Here we evaluated the impact that JGI trimming and decontamination procedures had on assembly and binning metrics, placement of metagenome assembled genomes into species trees, and functional profiles of metagenome-assembled genomes (MAGs) extracted from twenty three complex rhizosphere metagenomes. We also investigated how more aggressive trimming impacts these binning metrics. Results We found that JGI trimmed and decontamination of input reads had some significant impacts in assembly and binning metrics compared to raw reads, and that differences in placement of MAGs in species trees increased with decreasing completeness and contamination thresholds. More aggressive trimming beyond those used by JGI were found to reduce MAG counts. Conclusions Mild trimming and decontamination of metagenomics reads prior to assembly can change an investigator’s answer to the questions, “Who is there and what are they doing? However, mild trimming and decontamination of metagenomic reads with high quality scores is recommended for those who elect to do so.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2023

Explainable AI (XAI)-driven vibration sensing scheme for surface quality monitoring in a smart surface grinding process

Local Interpretable and Model-agnostic Explanation (LIME), an explainable artificial intelligence (XAI) approach is adapted to identify the globally important time-frequency bands for predicting average surface roughness (Ra) in a smart grinding process. The smart grinding setup consisted of a Supertech CNC precision surface grinding machine, instrumented with a Dytran piezoelectric accelerometer attached to the tailstock quill along the tangential direction (Y-axis). For every grinding pass, vibration signatures were captured, and the ground truth surface roughness values were recorded using a Mahr Marsurf M300C portable surface roughness profilometer. The roughness values ranged from 0.06 to 0.14 microns over the complete set of experiments. Time-frequency domain spectrogram frames were extracted for each of the vibration signals collected during the grinding process. Convolutional Neural Networks (CNNs) were modeled to predict the surface roughness based on these spectrogram frames and their image augmentations. The best CNN model was able to predict the roughness values with an overall R2-score of 0.95, training R2-score of 0.99, and testing R2-score of 0.81 with only 80 sets of vibration signals corresponding to 4 experiments with 20 trials each. Although the data size is not large enough to guarantee such performance metrics in real-world scenarios, one can extract statistically consistent explanations underlying the relationships these complex deep learning models capture. Further, the LIME methodology was implemented on the developed surface roughness CNN model to identify the important time-frequency bands (i.e., the superpixels of a spectrogram) influencing the predictions. Based on the identified important regions on the spectrogram frames, the corresponding frequency characteristics were determined that influence the surface roughness predictions. The important frequency range based on LIME results was approximately 11.7 to 19.1 kHz. The power of XAI was demonstrated by cutting down the sampling rate from 160 kHz to 30, 20, 10, and 5 kHz based on the important frequency range and considering Nyquist criteria. Separate CNN models were developed for these ranges by only extracting time-frequency contents below their corresponding Nyquist cut-offs. A proper data acquisition strategy is proposed by comparing the model performances to argue the selection of a sufficient sampling rate to capture the grinding process successfully and robustly.

42 ENGINEERING↗

DOE OSTI2023

Reprint of: Explainable AI (XAI)-driven vibration sensing scheme for surface quality monitoring in a smart surface grinding process

Local Interpretable and Model-agnostic Explanation (LIME), an explainable artificial intelligence (XAI) approach is adapted to identify the globally important time-frequency bands for predicting average surface roughness (Ra) in a smart grinding process. The smart grinding setup consisted of a Supertech CNC precision surface grinding machine, instrumented with a Dytran piezoelectric accelerometer attached to the tailstock quill along the tangential direction (Y-axis). For every grinding pass, vibration signatures were captured, and the ground truth surface roughness values were recorded using a Mahr Marsurf M300C portable surface roughness profilometer. The roughness values ranged from 0.06 to 0.14 microns over the complete set of experiments. Time-frequency domain spectrogram frames were extracted for each of the vibration signals collected during the grinding process. Convolutional Neural Networks (CNNs) were modeled to predict the surface roughness based on these spectrogram frames and their image augmentations. The best CNN model was able to predict the roughness values with an overall R2-score of 0.95, training R2-score of 0.99, and testing R2-score of 0.81 with only 80 sets of vibration signals corresponding to 4 experiments with 20 trials each. Although the data size is not large enough to guarantee such performance metrics in real-world scenarios, one can extract statistically consistent explanations underlying the relationships these complex deep learning models capture. Further, the LIME methodology was implemented on the developed surface roughness CNN model to identify the important time-frequency bands (i.e., the superpixels of a spectrogram) influencing the predictions. Based on the identified important regions on the spectrogram frames, the corresponding frequency characteristics were determined that influence the surface roughness predictions. The important frequency range based on LIME results was approximately 11.7 to 19.1 kHz. The power of XAI was demonstrated by cutting down the sampling rate from 160 kHz to 30, 20, 10, and 5 kHz based on the important frequency range and considering Nyquist criteria. Separate CNN models were developed for these ranges by only extracting time-frequency contents below their corresponding Nyquist cut-offs. A proper data acquisition strategy is proposed by comparing the model performances to argue the selection of a sufficient sampling rate to capture the grinding process successfully and robustly.

47 OTHER INSTRUMENTATION↗

DOE OSTI2025

JUSTIFI: Software for Improving Performance Objectives via Energy Efficiency

With growing energy supply concerns and rising costs, energy efficiency is a critical component of industrial energy resilience and competitiveness by directly reducing energy operating costs. Energy efficiency projects in manufacturing also yield valuable benefits to other key metrics, such as improved quality, reduced maintenance costs, improved safety, decreased pollution, and enhanced productivity. However, it is difficult to receive approval for energy efficiency projects, so implementation rates are low, even when meeting capital project payback period criteria. The inclusion and quantification of non-energy benefits (NEBs) in the decision-making process for energy efficiency projects can improve the overall financial payback period while demonstrating a positive impact on the firm's key performance metrics and business strategy. Despite their significant financial and strategic value, NEBs are rarely factored into decision-making due to lack of tools to effectively identify and quantify them. Therefore, a comprehensive and integrative approach is needed for the rapidly evolving energy landscape. To address these challenges, through funding from U.S. Department of Energy, our new assessment methodology integrates common continuous improvement six sigma concepts, such as the DMAIC process, and a protocol of guiding questions, into energy efficiency assessments to identify NEBs. We have also developed open-source software, JUSTIFI, to guide users through this process, data collection, and quantification. It is designed to be used concurrently with DOE energy system analysis software suite, MEASUR. Our methodology and tools inform energy assessors, firm engineering, decision makers, and workforce seeking to increase energy resilience and to maximize benefits aligned with performance metrics.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2026

JUSTIFI: Software for Improving Performance Objectives via Energy Efficiency

With growing energy supply concerns and rising costs, energy efficiency is a critical component of industrial energy resilience and competitiveness by directly reducing energy operating costs. Energy efficiency projects in manufacturing also yield valuable benefits to other key metrics, such as improved quality, reduced maintenance costs, improved safety, decreased pollution, and enhanced productivity. However, it is difficult to receive approval for energy efficiency projects, so implementation rates are low, even when meeting capital project payback period criteria. The inclusion and quantification of non-energy benefits (NEBs) in the decision-making process for energy efficiency projects can improve the overall financial payback period while demonstrating a positive impact on the firm's key performance metrics and business strategy. Despite their significant financial and strategic value, NEBs are rarely factored into decision-making due to lack of tools to effectively identify and quantify them. Therefore, a comprehensive and integrative approach is needed for the rapidly evolving energy landscape. To address these challenges, through funding from U.S. Department of Energy, our new assessment methodology integrates common continuous improvement six sigma concepts, such as the DMAIC process, and a protocol of guiding questions, into energy efficiency assessments to identify NEBs. We have also developed open-source software, JUSTIFI, to guide users through this process, data collection, and quantification. It is designed to be used concurrently with DOE energy system analysis software suite, MEASUR. Our methodology and tools inform energy assessors, firm engineering, decision makers, and workforce seeking to increase energy resilience and to maximize benefits aligned with performance metrics.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗